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ABSTRACT 


Force  Transformation  requires  a  much  greater  emphasis  on  testing  joint 
warfighting  capabilities.  A  unique  challenge  in  assessing  the  effectiveness  and  suitability 
of  systems  in  the  joint  environment  is  the  multitude  of  possible  interactions  and  outcomes 
in  a  system-of-systems  construct.  Because  of  resource  constraints  and  the  complexity  of 
conducting  live,  virtual,  and  constructive  testing  in  a  joint  mission  environment,  the  Joint 
Test  and  Evaluation  Methodology  (JTEM)  program  is  interested  in  determining  if 
analytical  techniques,  like  Modeling  and  Simulation,  can  be  applied  to  understand  the 
relationship  between  system-of-systems  performance  and  joint  mission  effectiveness.  As 
a  proof  of  concept,  a  Network  Enabled  Weapon  (NEW)  was  chosen  as  a  framework  for 
this  study.  This  thesis  uses  an  agent-based  distillation,  which  is  a  type  of  computer 
simulation,  to  model  the  critical  factors  of  interest  in  a  NEW  engagement  without 
explicitly  modeling  all  of  the  physical  details.  Using  cutting-edge  experimental  design 
techniques,  the  computer  model  was  run  many  tens  of  thousands  of  times,  with  the  results 
being  analyzed  to  determine  the  critical  parameters  required  for  mission  success.  The 
analysis  determined  key  interactions  in  NEW  system  performance  and  provides  JTEM 
with  a  framework  for  efficiently  conducting  testing  in  a  live  environment.  Specifically, 
the  results  indicate  sensor  range  of  a  third-party  ground  controller,  target  speed,  NEW 
impact  radius,  and  weapon  accuracy  as  the  key  factors  affecting  system  performance. 
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THESIS  DISCLAIMER 


The  reader  is  cautioned  that  the  computer  model  presented  in  this  research  may 
not  have  been  exercised  for  all  cases  of  interest.  While  every  effort  has  been  made, 
within  the  time  available,  to  ensure  that  the  programs  are  free  of  computational  and 
logical  errors,  they  cannot  be  considered  validated.  Any  application  of  these  programs 
without  additional  verification  is  at  the  risk  of  the  user. 

Additionally,  the  intent  of  this  effort  was  to  gain  insight  into  factors  affecting 
overall  system  performance.  The  research  was  not  conducted,  nor  was  the  model 
designed,  to  prove  or  demonstrate  the  capabilities  of  specific  weapon  systems  used  as  the 
framework  for  characteristics  of  the  simulated  entities.  As  such,  the  reader  should  not 
use  this  research  as  a  measure  of  assessing  the  effectiveness  or  suitability  of  weapon 
systems  currently  under  development. 
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EXECUTIVE  SUMMARY 


Joint  operations  have  become  the  mainstay  of  warfighting.  Force  Transformation 
requires  the  Test  and  Evaluation  (T&E)  community  to  place  a  much  greater  emphasis  on 
testing  joint  warfighting  capabilities.  A  unique  challenge  in  assessing  the  effectiveness 
and  suitability  of  systems  in  the  joint  environment  is  the  multitude  of  possible 
interactions  and  outcomes  in  a  system-of-systems  construct.  New  and  developing 
acquisition  programs  rely  on  interfaces  with  existing  or  future  systems,  quite  possibly 
from  separate  services,  to  achieve  mission  success.  Because  of  resource  constraints  and 
the  complexity  of  conducting  live,  virtual,  and  constructive  testing  in  a  joint  mission 
environment,  the  Joint  Test  and  Evaluation  Methodology  (JTEM)  program  is  interested  in 
determining  if  analytical  techniques,  like  Modeling  and  Simulation  (M&S),  can  be 
applied  to  understand  the  relationship  between  system-of-systems  performance  and  joint 
mission  effectiveness.  As  a  proof  of  concept  for  investigating  this  possibility,  Network 
Enabled  Weapons  (NEWs)  was  chosen  as  a  framework  for  further  study.  The  NEW 
concept  centers  on  the  ability  to  identify,  engage,  and  attack  moving  targets,  within 
moments  of  their  identification,  through  the  use  of  in-flight  target  updates  (IFTUs)  across 
a  Weapon  Control  Network  (WCN). 

As  acquisition  systems  like  NEW  are  required  to  conduct  more  testing  in  the 
context  of  a  joint  mission,  it  will  be  essential  that  these  tests  be  as  efficient  and  useful  as 
possible.  With  the  complexity  of  the  joint  test  environment,  M&S  is  one  of  the  most 
effective  tools  to  help  understand  the  environment,  design  an  efficient  and  useful  test,  and 
to  help  investigate  robust  possibilities  in  the  use  of  forces  to  accomplish  operational 
tasks.  This  thesis  used  an  agent-based  distillation,  which  is  a  type  of  computer 
simulation,  to  model  the  critical  factors  of  interest  in  combat  without  explicitly  modeling 
all  of  the  physical  detail.  Agent-Based  Modeling  (ABM)  refers  to  a  type  of  simulation 
made  up  of  agents  (or  entities)  that  behave  autonomously.  These  agents  possess  simple 
internal  rule-sets  for  decision  making,  movement,  and  action.  When  combined  with  other 
entities  in  the  model  and  subjected  to  stochastic  conditions,  the  agents  interact  in  ways 


xix 


that  are  often  reflective  of  large-scale  system  behavior.1  In  close  coordination  with 
ITEM  and  program  offices  responsible  for  networked  weapons,  the  author  and  the  model 
developer  worked  through  numerous  iterations  of  programming  and  debugging  over  the 
course  of  six  months  in  order  to  refine  the  model  and  improve  its  validity. 

As  with  many  complex  endeavors,  military  conflicts  typify  an  environment  of 
autonomous  or  semiautonomous  agents,  uncertainty  in  behavior  and  outcomes,  a  wide 
range  of  operational  inputs,  and  complex  interactions  between  entities.2  The  combination 
of  ABM  with  Data  Farming  offers  an  exploratory,  analytical  approach  to  broadly 
consider  uncertainties  associated  with  elements  of  warfare  that  might  otherwise  be  too 
costly  or  time  intensive  to  study  with  other  means. 

Data  Farming  involves  the  exploration  of  simulation  models  across  a  wide  range 
of  settings  for  the  agents’  characteristics  and  running  the  model  a  statistically  significant 
number  of  times.3  Rather  than  taking  a  “trial  and  error”  approach  to  experimental  design 
(be  it  live  or  in  the  M&S  environment),  researchers  often  use  specialized  techniques  to 
organize  the  myriad  of  possible  parameter  settings.  The  overall  objective  of  the  design  is 
to  maximize  the  information  gained  from  a  limited  number  of  experimental  runs.  In  his 
Ph.D.  Dissertation  at  the  Naval  Postgraduate  School,  FTC  Thomas  Cioppa,  USA, 
provides  an  approach  to  experimentation  geared  toward  addressing  the  issue  of  efficient 
design  and  analysis  techniques  for  complex  simulation  models.  His  research  indicates 
that  the  use  of  orthogonal,  or  nearly  orthogonal,  Fatin  hypercubes  with  excellent  space¬ 
filling  properties  enable  efficient  exploration  of  simulation  models.4  Unlike  traditional 
factorial  designs,  which  test  only  a  few  factors  at  a  minimum  number  of  levels,  a  space¬ 
filling  design  explores  a  broad  landscape  of  factor  settings,  as  indicated  in  Figure  ES-1. 


1  Susan  M.  Sanchez  and  Thomas  W.  Lucas,  “Exploring  the  World  of  Agent-Based  Simulations: 

Simple  Models,  Complex  Analyses,”  Proceedings  of  the  2002  Winter  Simulation  Conference, 

December  2002,  p.  1 . 

2  Thomas  M.  Cioppa,  Thomas  W.  Lucas,  and  Susan  M.  Sanchez,  “Military  Applications  of  Agent- 
Based  Simulations,”  Proceedings  of  the  2004  Winter  Simulation  Conference,  December  2004,  p.  1 . 

3  Gary  Home  and  Ted  Meyer,  “Data  Farming:  Discovering  Surprise,”  Proceedings  of  the  2004  Winter 
Simulation  Conference,  December  2004,  p.  2. 

4  Thomas  M.  Cioppa,  LTC,  USA,  “Efficient  Nearly  Orthogonal  and  Space-Filling  Experimental 
Designs  for  High-Dimensional  Complex  Models,”  Ph.D.  Dissertation,  Naval  Postgraduate  School, 
Monterey,  CA,  September  2002,  p.  9. 


XX 


20 

10 

200 

180 

160 

140 

25 

15 

5 

3000 

2000 

1000 

4 

2 

0 

250 

220 

190 

160 

60000 

40000 

20000 

200 

100 


Figure  ES-1.  Scatterplot  Matrix  for  a  Selection  of  Eight  Factors  Exhibiting  Excellent 

Space-Filling  Properties  of  the  Design. 

Figure  ES-1  shows  a  two-dimensional  projection  for  eight  of  the  factor 
combinations  in  the  single  target  (21 -factor)  design  and  exhibits  outstanding  space-filling 
properties.  As  a  separate  indicator  of  design  efficacy,  the  orthogonality  of  the  design 
improves  the  analysis  of  model  results  by  decreasing  correlation  among  regression 
coefficient  estimates,  and  by  allowing  investigation  of  multiple  high-order  interactions, 
nonlinear  relationships,  and  discontinuities  in  the  response.5 
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5  Thomas  M.  Cioppa,  LTC,  USA,  “Efficient  Nearly  Orthogonal  and  Space-Filling  Experimental 
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Monterey,  CA,  September  2002,  p.  21. 
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For  seven  factors,  this  design  approach  allows  the  exploration  of  model  effects 
across  17  uniformly  distributed  factor  settings,  in  as  few  as  17  model  runs.  In 
comparison,  a  three-level,  full  factorial  design  of  seven  factors  requires  37,  or 
2,187  model  runs,  and  would  only  examine  three  levels  of  each  factor.  When  considering 
that  this  research  examined  the  effects  and  interactions  of  more  than  20  factors  in  each 
design,  the  significance  is  the  difference  between  running  100  replications  on  a  single 
laptop  computer  in  five  hours,  or  waiting  until  the  sun  bums  out  and  never  seeing 
the  results. 

Using  these  cutting-edge,  experimental  design  techniques,  the  computer  model 
was  run  many  tens  of  thousands  of  times,  with  each  parameter  having  257  settings  varied 
uniformly  over  operationally  viable  ranges.  The  results  were  analyzed  to  determine  the 
critical  parameters  required  for  mission  success.  In  the  case  of  a  single  moving  target, 
indicative  of  a  wheeled  or  tracked  vehicle,  the  analysis  indicates  a  significant 
time-distance  interaction  between  the  sensor  range  of  the  ground-based  Joint  Terminal 
Attack  Controller  (JTAC)  and  the  speed  of  the  target.  Specifically,  when  the  target  speed 
is  less  than  13.2  meters/sec  (approximately  30  mph)  and  the  JTAC  sensor  range  exceeds 
2,117  meters,  the  model  indicates  an  80%  improvement  in  target  kills,  regardless  of  the 
other  parameter  settings.  Moreover,  when  a  combination  of  these  two  parameters  is 
constrained  across  a  realistic,  but  time-sensitive  range,  the  model  indicates  that  the 
amount  of  time  taken  by  the  decision  authority  to  issue  a  Close  Air  Support  (CAS) 
request  and  the  speed  at  which  the  launch  aircraft  flies  to  engage  the  target  provide  the 
most  improvement  in  mission  success. 

This  result  is  illustrated  through  use  of  a  partitioning  tree  in  Figure  ES-2.  The 
partitioning  (or  regression)  tree  is  a  statistical  analysis  approach  aimed  at  identifying  the 
most  critical  factors  affecting  the  response  variable — in  this  case,  the  proportion  of  target 
kills.  Each  block  lists  the  factor  name,  the  total  number  of  data  points  contained  in  the 
branch,  and  the  mean  value  and  standard  deviation  of  those  points.  To  perform  each 
“split”  of  the  tree,  the  statistical  software  recursively  divides  the  factor  that  most 
significantly  separates  the  means  by  examining  the  differences  in  sums  of  squares.  As 
shown  in  the  figure,  the  ratio  of  JTAC  sensor  range  to  target  speed  is  the  most  critical 


factor  in  the  model  for  the  first  two  splits  (albeit  at  different  threshold  values).  If  the  ratio 
is  between  197  and  305  seconds,  then  Command  and  Control  decision  time  and  aircraft 
speed  account  for  the  next  significant  splits,  at  the  values  shown.  It  should  be  noted  that 
while  the  ratio  variable  of  range  to  speed  provides  a  meaningful  way  to  conduct  the 
present  analysis,  the  variable  itself  is  notional  in  the  sense  that  it  depends  on  the  context 
in  which  the  specific  scenario  was  developed. 


Figure  ES-2.  Partition  Tree  for  the  Single  Mobile  Target  Scenario  with  Number  of  Kills 
as  the  Response  and  a  New  Explanatory  Factor  Created  from  Dividing  JTAC  Sensor 

Range  by  the  Target  Speed. 

To  test  the  system’s  ability  to  engage  a  subsequently  identified  high-value  target 
(HVT) — possibly  with  a  redirect  of  a  NEW  already  in  flight — scenarios  were  run  with 
two  targets  in  the  battlespace.  The  second  target  possesses  the  characteristics  of  a 
dismounted  individual  on  patrol.  The  model  indicates  different  results  for  the  kill  rate  of 
the  HVT,  depending  on  whether  or  not  the  launch  aircraft  contains  a  load  of  two  NEWs 
or  just  one.  In  the  case  of  one  weapon,  for  a  JTAC  sensor  range  of  less  than  two 
kilometers,  the  kill  rate  of  the  HVT  is  shown  to  improve  by  nearly  82%  if  the  IFTU 
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interval  is  less  than  50  seconds.  In  the  case  where  two  weapons  are  available,  the  most 
important  factors  affecting  the  HYT  kill  rate  are  specific  to  the  weapon  itself.  Namely,  if 
the  impact  radius  exceeds  5.4  meters  and  the  probability  of  kill  for  a  target  within  the 
blast  radius  is  greater  than  0.92,  then  the  overall  kill  rate  approaches  95%. 

While  relatively  simple  in  design,  the  simulation  model  provides  a  realistic 
depiction  of  operational  scenarios  and  the  interactions  of  systems  within  the  NEW 
construct.  Over  the  course  of  this  research,  the  author  consulted  with  JTEM  personnel 
and  subject  matter  experts  within  the  NEW  development  community  to  discuss  model 
functionality  and  the  ranges  of  settings  for  model  parameters.  The  author  and  the  model 
co-developer  worked  through  numerous  iterations  of  programming  and  debugging  over 
the  course  of  six  months  in  order  to  refine  the  model  and  improve  its  performance. 

The  results  of  the  analysis  determined  key  interactions  in  NEW 
system-of-systems  performance.  Additionally,  when  considered  in  context  of  the 
scenarios  developed,  the  model  provides  insight  for  program  managers  trying  to 
understand  the  required  performance  characteristics  of  systems  in  development.  Most 
importantly,  the  research  indicates  that  ABM,  especially  when  combined  with  efficient 
design  principles,  can  yield  a  method  to  quickly  analyze  a  complex  system-of-systems 
construct  and  provide  JTEM  with  a  framework  for  effectively  conducting  testing  in  a 
live  environment. 
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I. 


INTRODUCTION 


We  must,  therefore,  be  confident  that  the  general  measures  we  have 
adopted  will  produce  the  results  we  expect. 


-Karl  von  Clausewitz 


A.  BACKGROUND 

Joint  Vision  2020  was  published  by  the  Joint  Chiefs  of  Staff  to  describe  the 
operational  concepts  necessary  to  achieve  success  in  our  Nation’s  future  conflicts.  An 
uncertain  future  creates  a  broad  range  of  threat  possibilities  and  requires  an  investment  in 
technologies  and  new  military  capabilities.6 

The  Secretary  of  Defense’s  Quadrennial  Defense  Review  ( QDR ),  published 
30  September  2001,  indicates  that  the  transformation  required  to  meet  future  threats  is 
neither  an  end  point  itself,  nor  a  single  weapon  system  or  strategy.  Rather,  it  is  the 
process  used  to  achieve  warfighting  capabilities  and  is  based  on  the  following  four 
Transformational  Pillars: 

Strengthening  joint  operations  through  standing  joint  task  force 
headquarters,  improved  joint  command  and  control,  joint  training,  and 
expanded  joint  force  presence  policy; 

Experimenting  with  new  approaches  to  warfare,  operational  concepts  and 
capabilities,  and  organizational  constructs  such  as  standing  joint  forces 
through  wargaming,  simulations  and  field  exercises  focused  on  emerging 
challenges  and  opportunities; 

Exploiting  U.S.  intelligence  advantages  through  multiple  intelligence 
collection  assets,  global  surveillance  and  reconnaissance,  and  enhanced 
exploitation  and  dissemination; 


6  Joint  Chiefs  of  Staff,  Joint  Vision  2020,  United  States  Department  of  Defense,  Washington,  D.C., 
June  2000,  p.  1. 
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Developing  transformational  capabilities  through  increased  and  wide- 
ranging  science  and  technology,  selective  increases  in  procurement,  and 
innovations  in  DoD  processes.7 

In  line  with  the  pillar  of  experimentation,  the  QDR  further  states:  “While 
transformation  offers  U.S.  forces  the  promise  of  revolutionary  capabilities,  the  products 
of  this  transformation  must  be  tested  thoroughly  before  they  are  deployed.”8  In  his 
National  Military  Strategy,  the  Chairman  of  the  Joint  Chiefs  of  Staff  (CJCS)  indicates 
that  for  the  United  States  to  maintain  our  advantage  on  the  battlefield,  we  must  transform 
our  military  by  “combining  technology,  intellect  and  cultural  changes  across  the  joint 
community.”9  The  roadmap  for  the  future  of  the  U.S.  military  is  clearly  one  requiring  a 
transformation  of  capability  and  a  need  to  test  our  ability  to  achieve  success  in  a 
joint  environment. 

Complexity  arises  because  the  systems  participating  in  a  joint  mission  will 
interact  and  evolve  in  ways  that  are  not  easily  predictable.  Systems  are  designed  to  meet 
specific  requirements.  Developmental  testing  and,  to  a  degree,  operational  testing,  are 
designed  to  evaluate  whether  the  systems  can  meet  those  requirements.  As  those  systems 
are  given  to  operators  for  training  and  employment,  they  will  find  better  ways  to  use  the 
system’s  inherent  capabilities.  Operators  will  also  find  ways  to  work  around  capability 
limitations.  Tactical  decision  makers  will  employ  those  systems  in  combinations  and  in 
conditions  not  anticipated  by  the  designers  and  testers — often  in  response  to  changes  in 
tactics  of  an  evolving,  adaptive  adversary.  This  is  especially  true  when  different  systems, 
with  different  capabilities,  are  employed  in  a  joint  environment. 

B.  TEST  AND  EVALUATION  IN  A  JOINT  ENVIRONMENT 

Joint  operations  have  become  the  mainstay  of  warfighting.  Force  Transformation 
requires  the  Test  and  Evaluation  (T&E)  community  to  place  a  much  greater  emphasis  on 

7  United  States  Department  of  Defense,  Quadrennial  Defense  Review  Report,  Washington,  D.C., 

30  September  2001,  p.  32. 

8  Ibid.,  p.  41. 

9  Chairman  of  the  Joint  Chiefs  of  Staff,  National  Military  Strategy  of  the  United  States  of  America, 
United  States  Department  of  Defense,  Washington,  D.C.,  2004,  p.  15. 
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testing  joint  warfighting  capabilities  developed  in  response  to  the  Joint  Capabilities 
Integration  and  Development  System  (JCIDS)  process.  T&E  must  ensure  that  our 
combatant  commanders  (CoComs)  can  rely  on  equipment  (existing  and  future)  to  operate 
together  effectively  without  introducing  problems  to  the  warfighters.10 

Conducting  live  testing  is  further  complicated  by  the  limitations  of  defense 
budgets.  Of  the  over  $439  billion  in  the  2007  Department  of  Defense  (DoD)  budget,  a 
significant  portion,  about  $73  billion  is  devoted  to  Research,  Development,  Test,  and 
Evaluation  (RDT&E).  Only  $181  million,  about  one-quarter  percent  of  the  RDT&E 
amount,  however,  is  devoted  to  Operational  Test  and  Evaluation  (OT&E).  This  amplifies 
the  fact  that  operational  testing  must  be  structured  as  efficiently  as  possible.11 

In  2003,  the  Defense  Science  Board  Task  Force  Report  on  Enabling  Joint  Force 
Capabilities  stated: 

...  a  network-centric  approach  based  on  a  jointly  developed  network 
architecture  remains  essential  if  we  are  to  field  forces  that  can  (1)  respond 
quickly  to  a  wide  range  of  contingency  demands  and  (2)  act  decisively 
from  the  outset  against  adaptive  and  resourceful  adversaries.  This  requires 
full-capability,  highly  integrated  joint  land,  sea,  air,  and  space  forces.12 

Figure  1  depicts  the  flow  of  guidance  leading  to  the  need  for  integrated  testing  in 
the  joint  environment.  The  Transformation  Planning  Guidance,  published  by  the  United 
States  Secretary  of  Defense  (SecDef)  in  April  2003,  states: 

As  the  Department  transforms  to  a  joint  concept-centric  approach  for 
operational  planning  and  capabilities  development,  we  need  integrated 
architectures  that  define  the  specific  parameters  of  the  requisite  joint 
capabilities.  A  Joint  Test  and  Evaluation  Capability  is  needed  to  test  the 


10  Director,  Operational  T&E,  Testing  in  a  Joint  Environment  Roadmap  Strategic  Planning  Guidance, 
United  States  Department  of  Defense,  Final  Report,  November  2004,  p.  vii. 

11  DoD  budget  data  obtained  from  www.defenselink.mil/comptroller/defbudget/fy2007/index.html; 
site  last  accessed  in  May  2007. 

12  Defense  Science  Board,  Report  of  the  DSB  Task  Force  on  Enabling  Joint  Force  Capabilities, 
August  2003,  p.  1. 
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capabilities  in  a  realistic  joint  environment.  Test  and  evaluation  in  a  joint 
context  will  reveal  whether  or  not  the  integrated  architectures  present  a 
viable  application  of  warfighting  capabilities.13 


Figure  1 .  Building  the  Need  for  T&E  Transformation14 


The  Strategic  Planning  Guidance  in  March  2004  further  delineated  the  need  for  a 
transformation  in  T&E  and  instructed  the  Director,  Operational  Test  and  Evaluation 
(DOT&E),  to  conduct  a  study  to  develop  a  roadmap  on  the  way  ahead  for  T&E 
transformation.15  This  roadmap  was  signed  and  published  in  November  2004,  and  it 
addressed  three  major  issues:  (1)  The  need  to  construct  a  joint  environment  suitable  for 
testing  applications  from  the  mission  requirements  defined  by  the  JCIDS  capability 
planning  process;  (2)  the  requirement  to  use  this  environment  across  the  entire  acquisition 


13  Secretary  of  Defense,  Transformation  Planning  Guidance,  United  States  Department  of  Defense, 
Washington,  D.C.,  April  2003,  p.  20. 

14  Eileen  Bjorkman,  Col,  USAF,  “JTEM  Overview  Briefing,”  March  2006,  slide  3. 

15  Strategic  Planning  Guidance  for  FYs  2006-2011,  CLASSIFIED;  not  releasable  to  the  public. 
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process,  not  as  an  additional  test  requirement;  and  (3)  the  required  infrastructure,  to 
include  network  connectivity,  service-environments,  and  program-specific  modeling 
and  simulation.16 

C.  JOINT  TEST  AND  EVALUATION  METHODOLOGY  TEST  CASE 

Chartered  by  DOT&E  with  accomplishing  the  task  of  putting  the  roadmap  into 
action,  the  Joint  Test  and  Evaluation  Methodology  (JTEM)  program  was  established  in 
January  2006.  Their  overall  goal  is  to  develop  a  consistent  approach  to  T&E  in  the  joint 
environment.  Among  their  main  challenges  are  that  processes  and  methods  for  designing 
and  executing  tests  of  system-of-systems  are  not  well  defined  or  understood,  nor  is  there 
a  clear  understanding  of  how  to  assess  system  performance  pertaining  to  capabilities 
supporting  joint  missions.17  Due  to  resource  constraints  and  the  complexity  of 
conducting  live,  virtual,  and  constructive  testing  in  a  joint  mission  environment,  JTEM  is 
interested  in  determining  if  analytical  techniques,  like  modeling  and  simulation,  can  be 
applied  to  understand  the  relationships  between  system-of-systems  performance  and  joint 
mission  effectiveness.  As  a  proof  of  concept  for  investigating  this  possibility,  the 
Network  Enabled  Weapon  (NEW)  was  chosen  as  a  framework  for  further  study. 

The  NEW  concept  was  bom  out  of  the  requirement  to  create  a  network-centric 
integrated  system.  The  scope  of  this  research  effort  will  be  restricted  to  a  test  scenario 
for  the  current  NEW  concept  of  a  sub  500-lb  class,  nonpowered,  guided  bomb  with  data 
link  capabilities  and  several  guidance  modes  intended  to  attack  moving  targets  in  adverse 
weather  and  high-threat  environments  from  the  launch  platform. 

As  NEW  systems  are  required  to  conduct  more  testing  in  the  context  of  a  joint 
mission,  it  will  be  essential  that  these  tests  be  as  efficient  and  useful  as  possible.  With 
the  complexity  of  the  joint  test  environment,  Modeling  and  Simulation  (M&S)  is  one  of 


16  Director,  Operational  T&E,  Testing  in  a  Joint  Environment  Roadmap  Strategic  Planning  Guidance, 
United  States  Department  of  Defense,  Final  Report,  November  2004,  pp.  1-3. 

17  Eileen  Bjorkman,  Col,  USAF,  “JTEM  Overview  Briefing,”  March  2006,  slide  7. 
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the  most  effective  tools  to  help  understand  the  environment,  design  an  efficient  and 
useful  test,  and  to  help  investigate  robust  possibilities  in  the  use  of  forces  to  accomplish 
operational  tasks. 

This  thesis  will  use  an  agent-based  distillation,  which  is  a  type  of  computer 
simulation  that  is  used  to  model  the  critical  factors  of  interest  without  explicitly  modeling 
all  of  the  physical  details.  Chapter  II  provides  a  description  of  the  operational  system-of- 
systems  construct  and  a  discussion  on  model  development.  In  Chapter  III,  the 
experimental  design  methodology  will  be  introduced,  along  with  a  description  of 
scenarios  based  on  the  NEW  construct.  These  scenarios  will  be  replicated  in  the 
simulation  tool  and  then  analyzed.  The  analysis  process  will  use  a  technique  called  data 
farming.  This  involves  using  high-speed  computing  to  run  the  simulations  thousands  of 
times,  while  simultaneously  varying  many  input  parameters.  Using  cutting  edge 
experimental  designs  developed  at  the  Naval  Postgraduate  School,  the  data  resulting  from 
these  simulations  will  be  analyzed  and  presented  in  Chapter  IV  in  order  to  identify 
critical  factors,  interactions,  and  thresholds.  The  results  of  the  statistical  analysis  will 
then  be  extended  to  conclusions  in  Chapter  V  about  the  operational  context  and  used  to 
support  the  development  of  a  joint  test  plan  for  NEW  operations.  It  is  envisioned  the 
results  of  this  research  will  directly  support  the  planning  and  operation  of  JTEM  test 
events  for  NEW  operations — beginning  as  early  as  summer,  2007. 
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II.  SYSTEM  DESCRIPTION  AND  MODEL  DEVELOPMENT 


The  general  who  wins  a  battle  makes  many  calculations  in  his  temple 
before  the  battle  is  fought.  The  general  who  loses  a  battle  makes  but  few 
calculations  beforehand.  Thus  do  many  calculations  lead  to  victory,  and 
few  calculations  to  defeat:  how  much  more  no  calculation  at  all!  It  is  by 
attention  to  this  point  that  I  can  foresee  who  is  likely  to  win  or  lose. 

-Sun  Tzu,  The  Art  of  War 

A.  CHALLENGES  FOR  TESTING  IN  A  JOINT  MISSION  ENVIRONMENT 

A  unique  challenge  in  assessing  the  effectiveness  and  suitability  of  systems  in  the 
joint  environment  is  the  multitude  of  possible  interactions  and  outcomes  in  a 
system-of-systems  construct.  New  and  developing  acquisition  programs  rely  on 
interfaces  with  existing  or  future  systems,  quite  possibly  from  separate  services,  to 
achieve  mission  success. 

Focus  on  Joint  Mission  Effectiveness  (JMe)  has  increased  emphasis  on  joint  and 
coalition  forces,  integrated  operations,  network-enabled  warfare,  capabilities-based 
planning,  etc.  Test  and  Evaluation  (T&E)  to  ensure  that  systems  can  operate  in  this  new 
environment  thus  becomes  more  important,  and  increasingly  complex.  A  corollary 
requirement  is  the  analysis  of  JMe  to  assist  in  test  planning,  to  understand  and  quantify 
the  contribution  of  individual  systems  to  the  accomplishment  of  that  mission,  and  assess 
the  degree  to  which  the  mission  can  be  accomplished. 

Within  the  Joint  Mission  Environment  (JME),  existing  capability  gaps  impede 
successful  execution  of  Joint  Force  Commander  Joint  Targeting  Cycle  processes  against 
fixed,  mobile,  and  relocatable  planned  and  time-sensitive  targets  (TSTs).  Flexible  and 
responsive  network  operations  across  command,  control,  and  communications  systems, 
and  intelligence,  surveillance,  and  reconnaissance  infrastructure  are  critical  to 
successfully  prosecuting  these  targets  across  multiple  mission  areas. 
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The  concept  of  Precision  Engagement  typifies  this  system-of-systems  approach 
that  “enables  our  forces  to  locate  the  objective  or  target,  provide  responsive  command 
and  control,  generate  the  desired  effect,  assess  our  level  of  success,  and  retain  the 
flexibility  to  reengage  with  precision  when  required.”18 

B.  NETWORK  ENABLED  WEAPONS  (NEWs) 

Unlike  some  other  precision-guided  munitions  (PGMs),  the  emphasis  behind 
NEWs  is  the  ability  to  attack  stationary  and  mobile  targets  within  moments  of  their 
detection.  The  capability  to  engage  moving  targets  is  gained  through  use  of  an  onboard 
seeker  and  post-release  communications  network,  enabling  the  following  attributes: 

•  In-flight  target  updates  and  retargeting  provides  target  location  update 

•  Post-release  communications  acknowledgement  provides  positive 
weapon  control 

•  Weapon  in-flight  status  provides  feedback  of  weapon  location  and 
weapon  status 

•  Weapon  abort  provides  a  weapon  disabling  feature 

•  Bomb  impact  assessment  provides  pre-impact  target  damage 
information19 

NEWs  achieves  capability  against  mobile  targets  through  updates  from  the  launch 
aircraft  or  third-party  targeting  system  via  a  weapon  control  network  (WCN). 

The  base  network  configuration  for  this  research  will  include  three  active  nodes — 
the  strike  aircraft,  the  weapon,  and  the  “weapon  controller.”  The  weapon  controller  can 
refer  to  either  the  strike  (i.e.,  launch)  aircraft  or  a  third-party  Joint  Terminal  Attack 
Controller  (JTAC).  A  multitude  of  possible  mission  scenarios  exist,  but  an  example  of  a 
particular  construct  is  provided  in  Figure  2.  The  lightning  bolts  represent  networked 
communications  between  the  entities  involved.  Through  communications  with  the 
Combined  Air  Operations  Center  (CAOC),  an  identified  target  gets  assigned  to  a  mission 

18  Joint  Chiefs  of  Staff,  Joint  Vision  2010,  United  States  Department  of  Defense,  Washington,  D.C., 
1997,  p.  21. 

19  Headquarters,  Air  Combat  Command,  “Joint  Enabling  Concept  for  SDB  II,”  Draft,  p.  7-1,  FOUO; 
not  releasable  to  the  public. 
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typical  of  the  concept  of  operations  for  a  NEW.  During  flight  of  the  weapon,  target 
location  updates  are  passed  to  the  guidance  system  of  either  the  aircraft  or  the  NEW  in 
order  to  maneuver  the  munition  to  the  target.  This  networked  capability  is  what  enables 
the  engagement  of  moving  targets  without  using  existing  technologies  like  laser-guided 
identification  or  camera-guided  flight. 


Figure  2.  Guiding  a  NEW  to  the  Target. 

C.  AGENT-BASED  MODELING  (ABM) 

One  approach  towards  gaining  knowledge  about  the  possible  interactions 
associated  with  the  NEW  system-of-systems  is  through  the  application  of  ABM  and 
Data  Farming.  ABM  refers  to  a  type  of  simulation  made  up  of  agents  (or  entities)  that 
behave  autonomously.  These  agents  possess  simple  internal  rule-sets  for  decision 
making,  movement,  and  action.  When  combined  with  other  entities  in  the  model  and 
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subjected  to  stochastic  conditions,  the  agents  interact  in  ways  that  are  often  reflective  of 
large-scale  system  behavior.20  Data  Fanning  involves  the  exploration  of  simulation 
models  across  a  wide  range  of  settings  for  the  agents’  characteristics  and  running  the 
model  a  statistically  significant  number  of  times.21 

In  their  report  to  JTEM,  researchers  for  Referentia  Systems,  Inc.22  described 
ABM  and  Data  Farming: 

ABM  can  be  viewed  as  a  mathematical  function;  for  a  unique  set  of  input 
(including  any  specific  settings  of  the  seeds  for  the  random  number 
generators  used)  there  is  only  one  output,  both  of  which  may  be 
multidimensional.  The  goal  of  Data  Farming  is  to  understand  that 
mapping,  which  may  be  very  non-linear,  to  explore  and  identify  the 
relationships  between  the  input  parameters  and  the  output,  to  identify 
unexpected  relationships,  and  to  discover  and  characterize  regions  of  the 
input  space  where  discontinuities  may  exist.23 

The  overall  objective  for  the  use  of  Data  Farming  within  JTEM  is  to  extract  as 
much  information  as  possible  from  the  models  for  each  test  scenario.  It  is  envisioned  that 
ABM  will  serve  as  a  tool  for  refining  the  test  space  of  possible  parameters  and  parameter 
settings  and  narrow  the  focus  for  actual  live  test  events.24 

There  have  been  significant  amounts  of  research  regarding  the  applicability  of 
ABM  in  the  military  context.25  As  with  many  complex  endeavors,  military  conflicts 
typify  an  environment  of  autonomous  or  semiautonomous  agents,  uncertainty  in  behavior 
and  outcomes,  a  wide  range  of  operational  inputs,  and  complex  interactions  between 


20  Susan  M.  Sanchez  and  Thomas  W.  Lucas,  “Exploring  the  World  of  Agent-Based  Simulations: 
Simple  Models,  Complex  Analyses,”  Proceedings  of  the  2002  Winter  Simulation  Conference,  December 

2002,  p.  1. 

21  Gary  Home  and  Ted  Meyer,  “Data  Farming:  Discovering  Surprise,”  Proceedings  of  the  2004 
Winter  Simulation  Conference,  December  2004,  p.  2. 

22  Researchers  and  analysts,  widely  regarded  as  pioneers  in  the  application  of  ABM  combined  with 
Data  Farming. 

23  Gary  Home,  Steve  Upton,  and  Lawton  elites,  Joint  Test  and  Evaluation  Data  Farming  Project: 
Initial  Feasibility  and  Framing,  Final  Report  under  Contract  N00164-05-D-6656  TO  005,  3 1  January  2007, 
p.  15. 

24  Ibid.,  p.  24. 

25  See  Recent  Theses  and  Papers  at  http://harvest.nps.edu;  last  accessed  in  April  2007. 
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entities.26  ABM  offers  an  exploratory  analytical  approach  to  broadly  consider 
uncertainties  associated  with  elements  of  warfare  that  might  otherwise  be  too  costly  or 
time  intensive  to  study  with  other  means. 

D.  MODEL  DEVELOPMENT 

Under  a  research  contract  sponsored  by  JTEM,  Referentia  Systems,  Inc. 
developed  an  ABM  for  the  system-of-systems  concept.  Built  in  the  MASON 
programming  language27  and  dubbed  “TheTester,”  the  model  contains  the  functionality 
desired  for  studying  the  complex  interactions  within  a  NEW  scenario.  In  close 
coordination  with  JTEM  and  the  program  offices  responsible  for  networked  weapons,  the 
author  and  the  model  developer  worked  through  numerous  iterations  of  programming  and 
debugging  over  the  course  of  six  months  in  order  to  refine  the  model  and  improve  its 
performance.  Due  to  the  speed  of  the  model  runs,  and  the  graphical  user  interface 
implemented  by  the  developer,  the  author  was  able  to  provide  timely  feedback  and 
corrective  suggestions  to  fix  output  anomalies.  The  types  of  agents  contained  in  the  final 
model  used  in  this  research  effort  include:28 

•  Mobile  and  stationary  targets  of  various  classifications 

•  Strike  aircraft 

•  NEWs 

•  JTAC — ground-based  and  analogous  Unmanned  Aerial  Vehicle  (UAV) 

•  WCN 

•  Command  and  Control  (C2)  Center29 

At  the  start  of  the  simulation,  stationary,  mobile,  or  pop-up  targets  are  placed  on  a 
three-dimensional  grid  through  use  of  an  input  file.  When  a  target  enters  the  sensor  range 
of  a  JTAC  agent,  a  stochastically  governed  detection  may  occur.  Upon  detection,  the 

26  Thomas  Cioppa,  Thomas  W.  Lucas,  and  Susan  M.  Sanchez,  “Military  Applications  of  Agent-Based 
Simulations,”  Proceedings  of  the  2004  Winter  Simulation  Conference,  December  2004,  p.  1. 

27  See  http://www.cs.gmu.edu/~eclab/projects/mason/;  last  accessed  in  April  2007. 

28  Each  of  these  agents  and  their  parameters  are  explained  in  further  detail  in  Chapter  III. 

29  Gary  Home,  Steve  Upton,  and  Lawton  elites,  Joint  Test  and  Evaluation  Data  Farming  Project: 

Initial  Feasibility  and  Framing,  Final  Report  under  Contract  N00164-05-D-6656  TO  005,  3 1  January  2007, 

p.  11. 
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JTAC  relays  a  close  air  support  (CAS)  request  through  the  C2  network  to  an  available 
strike  aircraft.  The  aircraft  proceeds  toward  the  target  coordinates  given  by  the  JTAC 
until  it  reaches  a  predetermined  launch  (i.e.,  stand-off)  distance.  During  flight,  the  strike 
aircraft  receives  target  location  updates  directly  from  the  JTAC. 

When  the  strike  aircraft  reaches  the  launch  distance,  the  NEW  is  released  if  the 
target  location  information  is  not  “stale.”  If  this  condition  is  not  satisfied,  the  strike 
aircraft  either  returns  to  base  or  remains  on  course  toward  the  last  known  target  position 
and,  after  a  set  interval,  attempts  another  launch. 

Upon  successful  launch  of  the  weapon,  the  NEW  receives  in-flight  target  updates 
(IFTU)  of  the  target’s  location  until  it  gets  within  internal  sensor  range  of  the  target.  At 
that  point,  the  weapon  proceeds  autonomously  towards  the  target,  in  accordance  with  the 
rule  set  for  its  parameter  settings.  At  any  point  during  the  pre-  or  post-launch  sequence, 
if  a  target  of  higher  “value”  (HVT)  is  sensed  by  the  JTAC,  then  the  HVT  becomes  the 
target  of  primary  interest.  If  post-launch,  the  JTAC  will  query  the  NEW  through  the 
WCN  to  determine  if  it  has  the  capability  to  reach  the  HVT.  If  it  does,  then  the  NEW 
will  redirect  its  course  and  attempt  to  destroy  the  HVT.  Otherwise,  the  NEW  proceeds 
toward  the  original  target.  In  either  case,  once  the  NEW  goes  into  autonomous  seeker 
mode,  the  JTAC  is  no  longer  required  to  provide  IFTUs,  and  therefore  continues  sensing 
for  new  targets — possibly  including  the  original  target  in  the  case  of  a  redirect. 

The  main  indicator  of  whether  the  engagement  achieved  the  desired  JMe  centers 
on  the  concept  of  probability  of  a  single-shot-kill.  In  other  words,  was  the  best  available 
target  destroyed  in  a  timely  manner  with  a  single  NEW?  As  will  be  explained  in  the  next 
chapter,  there  are  numerous  dynamics  affecting  the  outcome  for  each  set  of  factor 
settings.  The  goal  will  be  to  determine  which  of  those  factors  (and  factor  settings)  most 
influences  “success”  of  the  mission  in  the  model.  From  a  thorough  examination  of  the 
interactions  within  this  joint  system-of-systems  model,  it  will  be  possible  to  draw 
recommendations  regarding  the  best  approach  for  live  testing  of  a  NEW  mission — 
thereby  achieving  JTEM’s  objectives  for  this  research. 
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Chapter  III  provides  the  foundation  for  the  experimental  design,  as  well  as  a 
detailed  description  of  the  model  agents,  their  individual  parameters,  and  the  ranges  over 
which  the  parameters  are  varied.  It  concludes  with  a  discussion  of  the  model  output 
(i.e.,  measures  of  effectiveness)  and  how  that  output  will  be  used  to  address  the  key 
research  objectives  for  this  thesis. 
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III.  DESIGN  OF  EXPERIMENTS  (DOE) 


The  secret  of  all  victory  lies  in  the  organization  of  the  non-obvious. 

-Marcus  Aurelius 

Marcus  Aurelius,  Emperor  of  the  Roman  Empire  from  161-180  A.D.,  was 
considered  one  of  the  great  Stoic  philosophers.30  The  Stoic  mindset  of  “following  where 
reason  leads”  typifies  the  leadership  expressed  in  his  quote  above,  for  it  is  through  an 
investigation  of  the  potential  nonobvious  outcomes  of  decisions  that  allow  informed 
leaders  to  improve  their  likelihood  of  success. 

Leaders  in  the  Department  of  Defense  (DoD)  routinely  make  decisions  involving 
expenditures  of  billions  of  dollars  or  involving  life  and  death  situations  for  personnel 
under  their  command.  In  an  ideal  setting,  the  outcomes  of  those  decisions  would  be 
known  with  certainty.  In  reality,  however,  the  vast  number  of  factors  with  influence  on 
the  process  or  system  under  consideration  often  introduces  a  high  degree  of  uncertainty  in 
the  outcome.  To  reduce  the  uncertainty,  DoD  authorities  rely  on  experimentation  to  help 
understand  the  potential  impacts  of  their  decisions.  Modeling  and  Simulation  (M&S)  is 
commonly  used  to  replace  live  experimentation  when  real  world  systems  are  unavailable 
or,  as  is  often  the  case,  too  expensive  to  devote  to  testing.31 

A.  GENERAL  APPROACH 

Rather  than  taking  a  “trial  and  error”  approach  to  experimental  design  (be  it  live 
or  in  the  M&S  environment),  researchers  often  utilize  specialized  techniques  to  organize 
the  myriad  of  possible  parameter  settings.  The  overall  objective  of  the  design  is  to 
maximize  the  information  gained  from  a  limited  number  of  experimental  runs. 


30  For  more  on  Marcus  Arelius  and  Stoic  philosophy  see  http://en.wikipedia.org/wiki/Marcus_Aurelius 
and  http://en.wikipedia.org/wiki/Stoicism,  respectively.  Last  accessed  in  May  2007. 

31  Thomas  W.  Lucas  and  Susan  M.  Sanchez,  “The  Brave  New  World  of  Simulation  Experiments  for 
Defense  and  Homeland  Security  Applications,”  Proceedings  of  the  2006  Joint  Statistical  Meetings,  p.  1 . 
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In  his  Ph.D.  Dissertation  at  the  Naval  Postgraduate  School,  LTC  Thomas  Cioppa, 
USA,  provides  an  approach  to  experimentation  geared  toward  addressing  the  issue  of 
efficient  design  and  analysis  techniques  for  complex  simulation  models.  His  research 
indicates  that  the  use  of  orthogonal,  or  nearly  orthogonal,  Latin  hypercubes  with  excellent 
space-filling  properties  enable  efficient  exploration  of  simulation  models.32  Unlike 
traditional  factorial  designs,  which  test  only  a  few  factors  at  a  minimum  number  of  levels, 
a  space-filling  design  explores  a  broad  landscape  of  factor  settings.  The  orthogonality  of 
the  design  improves  the  analysis  of  model  results  by  decreasing  dependence  among 
regression  coefficient  estimates,  and  by  allowing  investigation  of  multiple  high-order 
interactions,  nonlinear  relationships,  and  discontinuities  in  the  response.33 

Figure  3  demonstrates  the  space-filling  properties  of  the  Nearly-Orthogonal  Latin 
Hypercube  (NOLH)  design.  The  figure  displays  the  pairwise  combinations  of  factor 
settings  in  a  two-dimensional  projection.  For  example,  the  highlighted  block  shows  the 
combination  of  factor  settings  this  experiment  would  test  for  Factor  B  and  Factor  D.  In 
three  dimensions,  the  image  would  portray  settings  for  three  factors  spaced  efficiently 
throughout  a  cube.  While  we  cannot  visually  comprehend  the  image  in  seven 
dimensions,  the  mathematics  and  the  concepts  still  hold  true.  For  seven  factors,  this 
design  approach  allows  the  exploration  of  model  effects  across  1 7  uniformly  distributed 
factor  settings,  in  a  total  of  as  few  as  17  model  runs.  In  comparison,  a  three-level,  full 
factorial  design  of  seven  factors  would  require  37,  or  2,187  model  runs,  and  would  only 
examine  three  levels  of  each  factor.  Considering  that  the  intent  is  to  replicate  each  design 
point  hundreds  or  thousands  of  times  to  draw  out  the  true  nature  of  the  stochastic 
response,  the  orders  of  magnitude  difference  in  the  number  of  model  runs  becomes 
increasingly  significant.  More  poignantly,  when  considering  that  this  research  will 
examine  the  effects  and  interactions  of  more  than  20  factors  in  each  design,  the 
significance  is  the  difference  between  running  100  replications  on  a  single  laptop 


32  Thomas  M.  Cioppa,  LTC,  USA,  “Efficient  Nearly  Orthogonal  and  Space-Filling  Experimental 
Designs  for  High-Dimensional  Complex  Models,”  Ph.D.  Dissertation,  Naval  Postgraduate  School, 
Monterey,  CA,  September  2002,  p.  9. 

33  Ibid.,  p.  21. 
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computer  in  five  hours,  or  waiting  until  the  sun  bums  out  (e.g.,  2017  x  100  runs)  and 
never  seeing  the  results — and  this  is  for  a  simulation  model  that  takes  less  than  one 
second  to  complete  each  run. 
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Figure  3.  NOLH  Scatterplot  for  7  Factors  and  17  Design  Points  Demonstrating 
Excellent  Space-Filling  Properties  of  the  Design. 


B.  MODEL  PARAMETERS 

Each  of  the  agents  described  in  Chapter  II  possess  one  or  more  characteristics  that 
define  their  behavior  within  the  model.  Depending  on  the  specific  scenario  and  the 
objectives  for  each  design,  some  of  the  characteristics  (synonymously  referred  to  as 
parameters  or  factors)  will  be  held  to  a  constant  value,  while  the  majority  are  varied  over 
a  predefined  range.  Unless  specifically  stated  otherwise,  the  factors  are  set  to  a  particular 
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value  for  each  model  run  and  do  not  fluctuate  during  that  particular  run.  The  following 
sections  describe  the  parameters  for  each  agent.  A  summary  description  is  provided  in 
Appendix  A. 

1.  Joint  Terminal  Attack  Controller  (JTAC) 

The  JTAC  is  typically  a  ground-based  controller  deployed  within  the  immediate 
vicinity  of  potential  or  known  targets.  He  possesses  sensor  capabilities,  communication 
devices,  and  equipment  necessary  to  locate,  identify,  and  track  targets.  Additionally,  he 
is  able  to  relay  target  location  information  to  other  agents  on  the  Weapon  Control 
Network  (WCN).  The  agent  could  be  used  to  replicate  an  air-based  controller,  such  as  an 
Unmanned  Aerial  Vehicle  (UAV),  by  adjusting  the  parameters  according  to  the 
characteristics  of  the  entity  being  modeled. 

a.  Speed 

If  necessary,  the  JTAC  can  be  assigned  waypoints  from  his  initial  location 
to  traverse  during  execution  of  the  scenario.  The  speed  setting  controls  how  fast  the 
JTAC  moves  between  the  assigned  waypoints.  Speed  is  measured  in  meters/second. 

b.  Sensor  Range 

The  JTAC  sensor  range  is  a  radial  distance  defining  a  circular 
representation  of  the  agent’s  ability  to  identify  and  track  targets.  The  sensor 
characteristics  are  commonly  referred  to  as  “cookie  cutter,”  indicating  a  defined 
probability  of  detection  for  targets  within  range  and  a  probability  of  zero  for  targets 
outside  the  range.  Sensor  range  is  measured  in  meters. 

c.  Probability  of  Detection  (Pd) 

Targets  within  the  JTAC’s  sensor  range  are  detected  according  to  the 
assigned  Pd.  During  each  model  time  step,  a  random  draw  determines  whether  or  not  the 
target  is  detected.  The  logic  within  the  model  can  be  adjusted  such  that  the  Pd  becomes 
one  (certainty),  after  initial  detection  occurs,  to  simulate  focused  attention  on  the  target. 
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d.  Multi-Target  Degradation  Factor  (MTDF) 

In  scenarios  containing  more  than  one  target,  the  JTAC’s  ability  to  detect 
multiple  targets  may  be  degraded  to  represent  the  difficulty  associated  with  simultaneous 
engagements.  The  MTDF  is  a  factor  between  0  and  1  applied  multiplicatively  to  the  Pd. 

e.  Target  Location  Error  (TLE) 

TLE  defines  the  accuracy  associated  with  the  JTAC’s  perception  of  the 
true  location  of  the  target.  It  is  represented  in  the  model  by  a  Bivariate  Normal 
distribution  and  is  defined  by  two  parameters,  the  mean  and  standard  deviation.34 

f  In-Flight  Target  Update  (IFTU)  Interval 

The  JTAC  does  not  continuously  provide  target  location  updates  to  other 
agents  on  the  network.  Instead,  updates  are  transmitted  at  a  frequency  defined  by  the 
assigned  IFTU — measured  in  seconds. 

2.  Network  Enabled  Weapon  (NEW) 

The  NEW  for  this  study  is  a  nonpowered  weapon  launched  from  an  aircraft.  The 
combination  of  networked  communications,  control  surfaces,  and  internal  guidance 
systems  give  it  the  ability  to  redirect  its  path  during  flight  and  engage  moving  targets. 
Upon  direction  of  the  JTAC,  the  NEW  is  also  capable  of  being  reassigned  to  a 
higher-value  target  (HVT)  during  flight.  Rather  than  modeling  the  complicated  physics 
associated  with  determining  a  nonpowered  weapon’s  ability  to  reach  a  new  target,  a 
simplified  approach  was  developed.  Upon  request  from  the  JTAC  for  a  redirect,  the 
model  calculates  the  three-dimensional  slant  distance  to  both  targets  and  adjusts  the 
weapon’s  path  only  if  the  HVT  is  closer  than  the  original  target.  In  the  case  where  the 


34  Jay  L.  Devore,  Probability  and  Statistics  for  Engineering  and  the  Sciences,  6th  Edition,  2004, 
Thomson  Brooks/Cole  Publishers,  2004,  p.  543. 
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weapon  is  not  able  to  reach  the  HVT,  a  message  is  transmitted  through  the  network  to  the 
JTAC  indicating  the  redirect  was  denied,  and  the  NEW  continues  flying  toward  the 
original  target. 


a.  Speed 

The  true  speed  of  flight  is,  again,  a  complicated  physical  process  to  model. 
The  simplifying  approach  for  this  study  is  to  vary  the  speed  as  a  parameter  across  a 
typical  range  between  maximum  glide  performance  and  minimum  time  to  target.  Every 
effort  was  made  to  consult  with  Subject  Matter  Experts  (SMEs)  in  order  to  accurately 
represent  weapon  performance.35  Speed  is  measured  in  meters/second. 

b.  Impact  Radius 

The  impact  radius  defines  a  circular  area  of  lethality  around  the  center  of 
the  bomb’s  impact  location.  Upon  impact,  the  model  determines  whether  the  target’s  true 
location  is  within  the  area  and,  in  combination  with  the  probability  of  kill  (Pk),  assigns  a 
target  kill  or  miss  accordingly.  The  impact  radius  is  measured  in  meters. 

c.  Sensor  Range 

The  sensor  range  identifies  the  point  at  which  the  NEW  acquires 
autonomous  recognition  of  the  target.  The  performance  within  the  model  is  similar  to  the 
“cookie  cutter”  approach  described  for  the  JTAC  agent.  Once  the  NEW  is  within  this 
three-dimensional  range  to  the  target,  it  no  longer  receives  IFTUs  through  the  network; 
instead  relying  on  its  own  sensor  characteristics  to  guide  it  to  the  target. 

d.  Target  Location  Error 

The  TLE  is  identical  to  the  process  described  for  the  JTAC.  The  Gaussian 
mean  and  standard  deviation  parameters,  however,  will  take  on  a  different  range  of 
values  in  order  to  more  closely  represent  NEW  sensor  capabilities. 

35  The  author  and  model  developer  consulted  in  person  with  SMEs  from  Joint  Command  and  Control 
for  NEW  and  SDB  II  program  offices,  Eglin  AFB,  on  26  April  2007. 
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e.  Probability  of  Detection  (Pd) 

Once  in  autonomous  seeker  mode,  the  NEW  relies  on  its  own  Pd  to 
determine  if  it  senses  the  target  during  a  particular  time  step.  Unlike  the  JTAC,  however, 
the  Pd  is  always  a  random  draw,  according  to  the  parameter  setting. 

f.  Probability  of  Kill  (Pk) 

Combined  with  the  impact  radius,  the  NEW’s  Pk  value  determines 
whether  or  not  a  target  is  killed.  The  Pk  is  limited  to  very  high  (i.e.,  favorable)  values  to 
replicate  the  likelihood  of  correct  weapon  detonation  upon  impact. 

3.  Target 

Targets  for  this  study  are  notional  in  that  they  do  not  have  any  distinguishing 
characteristics  to  differentiate  them  among  typical  NEW  target  sets.  Targets  can, 
however,  be  assigned  a  priority  value  in  order  to  account  for  the  presence  of  an  HYT. 
Also,  the  targets  may  be  assigned  different  Pd  and  Pk  values  for  the  JTAC  and  the  NEW. 
The  two  parameters  explicitly  varied  as  part  of  the  DoE  are  the  target  speed  and  the 
pop-up  time. 


a.  Speed 

For  each  scenario,  the  target(s)  are  assigned  predetermined  waypoints  to 
traverse.  The  target  may  be  set  to  cycle  through  the  waypoints  continuously  or  stop  when 
it  reaches  the  last  one.  The  target  speed  is  used  to  differentiate  between  categories  of 
targets.  For  example,  vehicular  targets  typically  travel  at  much  greater  speeds  than 
dismounted  (i.e.,  foot  patrol)  targets.  Speed  is  measured  in  meters/second. 

b.  Pop-Up  Time 

Each  target  in  the  scenario  may  be  designated  a  particular  time  step  in 
which  it  first  appears.  The  pop-up  time  presents  the  opportunity  to  investigate  system 
effectiveness  against  a  HVT  that  may  result  in  a  redirecting  a  NEW  already  in  flight. 
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4. 


Launch  Aircraft 


The  aircraft’s  initial  location  can  be  varied  within  the  model,  but  for  the  purposes 
of  this  study,  the  starting  position  is  simulated  as  a  Combat  Air  Patrol  (CAP)  orbit 
approximately  100  kilometers  (km)  south  of  the  JTAC  at  an  altitude  of  six  km.  After 
target  identification,  the  aircraft  agent  receives  an  assigned  mission  from  the 
Command  and  Control  (C2)  Center  and  is  loaded  with  a  predetermined  number  of  NEWs. 
The  aircraft  is  then  directed  toward  the  target  by  location  updates  relayed  directly  from 
the  JTAC.  In  accordance  with  the  parameter  settings  for  a  particular  model  run,  the 
aircraft  attempts  to  launch  the  NEW  and  then  proceeds  back  to  its  starting  position.  The 
aircraft  may,  if  equipped  with  remaining  weapons,  be  directed  at  any  point  to  conduct 
another  mission. 


a.  Speed 

The  aircraft  speed  setting  is  varied  over  a  range  representative  of  a  typical 
strike  aircraft  and  is  constant  throughout  a  particular  model  run.  Aircraft  speed  is 
measured  in  meters/second. 

b.  Launch  Distance,  Update  Requirement,  Number  of  Attempts, 
and  Reattempt  Interval 

This  section  describes  four  different  factors  for  the  aircraft  agent.  The 
launch  distance  represents  the  stand-off  distance  where  the  aircraft  will  initially  attempt 
to  release  the  NEW.  If  the  target  location  information  has  not  been  successfully  relayed 
to  the  aircraft  within  the  time  period  set  by  the  update  requirement  factor,  the  aircraft  will 
not  launch  the  weapon.  This  is  designed  to  limit  the  launch  of  a  weapon  when  the  target 
is  no  longer  within  sensor  range  of  the  JTAC.  Based  on  the  number  of  attempts  setting, 
the  aircraft  may  make  subsequent  attempts  to  launch  the  NEW,  at  a  frequency  governed 
by  the  reattempt  interval.  While  awaiting  multiple  attempts  to  launch  the  weapon,  the 
aircraft  continues  to  fly  toward  the  last  known  target  location. 
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5. 


Weapon  Control  Network  (WCN) 


The  WCN  is  the  host  for  the  flow  of  communication  between  the  NEW  and  JTAC 
agents.  It  is  the  heart  of  the  capability  to  operate  in  a  network-centric  environment. 
While  there  are  a  multitude  of  potential  issues  and  limitations  affecting  the  transmission 
capability  and  interoperability  of  systems  across  any  network,  the  model  used  for  this 
research  considers  only  two  of  the  important  factors.  The  range  of  the  network  is 
simulated  to  encompass  the  entire  range  of  the  battlespace. 

a.  Reliability 

The  reliability  of  the  network  is  a  parameter  that  governs  the  random 
probability  of  a  successful  message  transmission.  For  the  purposes  of  this  research, 
messages  transmitted  are  either  fully  received  or  not  at  all — there  are  no  instances  of 
partial  (or  garbled)  communications. 

b.  Latency 

Latency  is  a  measure  of  how  long  it  takes  for  a  message  to  be  transmitted 
across  the  network  and  is  measured  in  seconds. 

6.  Command  and  Control  (C2)  Center 

When  the  JTAC  senses  a  target,  the  information  is  transmitted  to  the  C2  Center  in 
the  form  of  a  Close  Air  Support  (CAS)  request.  The  intent  is  to  simulate  a  command 
element  within  the  scenario  with  the  authority  to  assign  an  aircraft  equipped  with  a  NEW 
to  prosecute  the  target. 

a.  Decision  Time 

The  only  parameter  currently  in  use  for  the  C2  agent  is  the  latency 
associated  with  the  amount  of  decision  time  it  takes  to  assign  the  mission  to  an  available 
aircraft.  The  time  is  measured  in  seconds  and  effectively  introduces  a  delay  in  between 
target  identification  and  CAS  mission  authorization. 
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C.  SCENARIO  DEVELOPMENT 

NEW  missions  typically  fall  into  the  category  of  “reactionary  strike,”  meaning 
targets  of  opportunity  that  involve  little  or  no  prior  knowledge  of  the  target’s  exact 
location  and/or  route.  The  missions  may  involve  moving  or  stationary  targets. 

This  research  considers  a  scenario  involving  ground-based  mobile  targets  and  a 
JTAC  as  the  post-launch  weapon  controller.  One  of  the  targets  is  envisioned  as  a 
wheeled  vehicle  that  follows  a  prescribed  route  within  the  vicinity  of  the  stationary 
JTAC.  Depending  on  the  settings  for  a  particular  model  run,  a  second  target  of  higher 
priority,  simulating  a  dismounted  combatant  may  “pop  up”  in  the  battlespace. 

As  depicted  in  Figure  4,  the  JTAC  is  located  at  the  center  of  the  simulation 
battlespace.  The  vehicular  target  begins  its  route  at  a  position  2,000  meters  east  and 
1,000  meters  north  of  the  JTAC.  The  target  then  follows  a  series  of  set  waypoints  at  its 
designated  speed.  At  some  point  in  the  scenario,  the  target  will  enter  (and  exit  if  not 
destroyed)  the  sensor  range  of  the  controller,  initiating  a  CAS  request  once  detected.  In 
some  scenarios,  an  HVT  may  appear  to  the  southeast  of  the  controller’s  position.  This 
target  will  cycle  through  a  square  pattern  at  a  speed  indicative  of  a  human  on  foot  patrol. 
Upon  detection  of  the  HVT,  the  JTAC  will  submit  a  new  CAS  request,  which  may  result 
in  either  an  attempt  to  redirect  an  inbound  NEW  or  the  launch  of  a  second  weapon, 
if  available. 
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Figure  4.  Visual  Depiction  of  the  Battlespace  Indicating  JTAC  Positioning 

and  Target  Routing. 

This  scenario  is  based,  in  part,  on  mission  descriptions  found  in  the  draft  Joint 
Enabling  Concept  document  for  SDB  II.36  The  intent  is  to  simulate  a  realistic  operational 
setting  that  would  offer  the  opportunity  to  test  the  NEW  system-of-systems’  effectiveness 
against  mobile  targets  and  time-sensitive  HVTs.  Several  effectiveness  measures  were 
developed  in  order  to  facilitate  the  assessment. 

D.  MEASURES  OF  EFFECTIVENESS  (MOEs) 

Because  of  resource  constraints  and  the  complexity  of  conducting  testing  in  a 
joint  mission  environment,  JTEM  is  interested  in  determining  if  analytical  techniques, 
like  modeling  and  simulation,  can  be  applied  to  understand  the  relationship  between 
system-of-systems  performance  and  Joint  Mission  Effectiveness  (JMe).  Specifically,  this 
research  was  designed  to  address  the  following  questions: 


3 ^  Headquarters,  Air  Combat  Command,  “Joint  Enabling  Concept  for  SDB  II,”  Draft,  Section  8, 
FOUO;  not  releasable  to  the  public. 
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•  For  a  particular  scenario,  what  are  the  potential  performance  parameters 
(and  their  applicable  ranges)  requiring  testing  to  determine 
system  capability? 

•  Given  that  the  ability  to  conduct  live  testing  is  resource  constrained,  what 
is  the  critical  collection  of  parameter  inputs  to  test? 

•  For  a  particular  construct,  are  there  possible  mixtures  of  operational  forces 
or  Tactics,  Techniques,  and  Procedures  (TTPs)  that  could  have  a 
significant  impact  on  JMe? 

•  Can  Agent-Based  Modeling  (ABM)  assist  JTEM  in  determining 
approaches  to  Joint  Test  and  Evaluation? 

The  main  measure  used  in  this  study  to  address  the  research  questions  is  the 
number  of  target  kills.  The  output  files  generated  by  the  model  provide  a  binary 
description  of  whether  a  particular  model  run  resulted  in  a  kill  for  each  target  type,  as 
applicable.  The  pertinent  output  data  generated  by  the  model  is  presented  in  Table  1. 


MOE 

Description 

Target  Kill 

Binary  kill  indicator  for  each  target  type  in  each  model  run 

CAS  Requests 

Total  number  of  CAS  requests  in  each  model  run 

NEW  Launches 

Total  number  of  NEW  launches  in  each  model  run 

Abort  -  No  Target  Update 

Mission  abort  -  Launch  aircraft  did  not  have  recent  target  info 

Redirected 

Redirect  of  the  NEW  occurred  during  mission 

Redirect  Denied 

Redirect  of  the  NEW  denied  during  the  mission 

Hit  -  No  Kill 

NEW  hit  within  impact  radius  of  target,  but  no  kill 

Outside  Impact 

NEW  missed  the  target 

Table  1.  Model  Output  Used  as  MOEs. 


E.  EXPERIMENTAL  DESIGNS 

The  designs  used  for  the  research  are  based  on  the  orthogonal  and  space-filling 
concepts  previously  discussed.  Two  NOLH  design  matrices37  were  constructed — one  for 
a  scenario  involving  the  vehicular  target  and  one  containing  parameters  for  both  the 
vehicular  and  pop-up  HVT.  Table  2  shows  the  parameter  settings  for  the  two  target 


37  The  spreadsheet  tool  used  to  build  the  designs  can  be  found  under  the  Software  Downloads  link  at 
http://harvest.nps.edu.;  last  accessed  in  April  2007. 
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scenario.  A  total  of  27  factors  were  varied  over  the  ranges  shown.  Several  factors,  such 
as  the  pop-up  time  for  the  Mobile- 1  target,  are  included  in  the  table  but  were  assigned  a 
specific  value  consistent  across  all  model  runs.  The  one  target  scenario  factor  ranges  are 
identical  to  those  shown  in  Table  2,  with  the  exception  of  the  six  highlighted  parameters 
unique  to  the  HVT  scenario,  which  were  excluded. 


Factor  Name 

Description 

Low  Value 

High  Value 

Units 

Tgt  A  Spd 

High-Value  Target  (HVT)  Speed 

0.5 

4 

meters/sec 

Tgt  A  PUT 

HVT  Pop-up  Time 

100 

600 

seconds 

Tgt  C  Spd 

Vehicular  Target  Speed 

4 

20 

meters/sec 

Tgt  C  PUT 

Vehicular  Target  Pop-up  Time 

10 

10 

seconds 

NEW  Spd 

NEW  Speed 

140 

200 

meters/sec 

NEW  ImpRad 

NEW  Impact  Radius 

3 

30 

meters 

NEW  SnsrRng 

NEW  Sensor  Range 

1000 

3000 

meters 

NEW  TLEmu 

NEW  Mean  Target  Location  Error  (TLE) 

0 

5 

meters 

NEW  TLEsigma 

NEW  TLE  Standard  Deviation 

0 

2 

meters 

NEW  PdA 

NEW  Probability  Detect  HVT 

0.85 

1.0 

n/a 

NEW  PdC 

NEW  Probability  Detect  Vehicular  Target 

0.85 

1.0 

n/a 

NEW  PkA 

NEW  Probability  Kill  HVT 

0.9 

1.0 

n/a 

NEW  PkC 

NEW  Probability  Kill  Vehicular  Target 

0.9 

1.0 

n/a 

AC  Spd 

Aircraft  Speed 

154 

254 

meters/sec 

AC  LnchDist 

Aircraft  Stand-off  Launch  Distance 

18.5 

74 

kilometers 

AC  UpdtReq 

Aircraft  Requirement  for  Target  Info  Recency 

10 

45 

seconds 

AC  LnchAtmpts 

Aircraft  Number  of  Launch  Attempts 

1 

5 

n/a 

AC  Lnchlntrvl 

Aircraft  Interval  Between  Launch  Attempts 

10 

30 

seconds 

JTAC  Spd 

JTAC  Speed 

0 

0 

meters/sec 

JTAC  SnsrRng 

JTAC  Sensor  Range 

500 

5000 

meters 

JTAC  MTDF 

JTAC  Multi-Target  Degradation  Factor 

0.5 

1.0 

n/a 

JTAC  TLEmu 

JTAC  Mean  TLE 

10 

100 

meters 

JTAC  TLEsigma 

JTAC  TLE  Standard  Deviation 

2 

20 

meters 

JTAC  PdA 

JTAC  Probability  Detect  HVT 

0.7 

1.0 

n/a 

JTAC  PdC 

JTAC  Probability  Detect  Vehicular  Target 

0.7 

1.0 

n/a 

JTAC  IFTU  Intrvl 

JTAC  In-flight  Target  Update  Interval 

15 

60 

secnds 

WCN  Rel 

Weapon  Control  Network  Reliability 

0.95 

1.0 

n/a 

WCN  Lat 

Weapon  Control  Network  Latency 

0 

3 

seconds 

C2  DecTime 

Command  and  Control  Decision  Time 

10 

200 

seconds 

Indicates  factors  not  used  in  the  one  target  scenario 

Table  2.  Factor  Settings  and  Ranges  for  the  Two-Target  Scenario. 

Figure  5  shows  a  two-dimensional  projection  for  eight  of  the  factor  combinations 
in  the  single  target  (21 -factor)  design.  As  previously  shown  in  the  example  at  Figure  3, 
the  design  exhibits  outstanding  space-filling  properties. 
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Figure  5.  Scatterplot  Matrix  for  a  Selection  of  Eight  Factors  Exhibiting  Excellent 

Space-Filling  Properties  of  the  Design. 

Table  3  shows  the  shows  the  correlation  matrix  for  the  same  eight  factors.  As 
indicated  in  Cioppa’s  research,  the  columns  of  the  design  matrix  are  “nearly  orthogonal,” 
exhibited  by  the  minimal  correlation,  in  this  case  much  less  than  +/-  0.03. 38  The  decision 
to  display  a  selection  of  only  8  of  the  21  factors  included  in  the  design  was  one  of 
presentation  and  space.  The  concept  and  the  design  results  extend  similarly  to  an 
analysis  of  the  complete  experimental  design  matrix. 
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38  Thomas  M.  Cioppa,  LTC,  USA,  “Efficient  Nearly  Orthogonal  and  Space-Filling  Experimental 
Designs  for  High-Dimensional  Complex  Models,”  Ph.D.  Dissertation,  Naval  Postgraduate  School, 
Monterey,  CA,  September  2002,  p.  23. 
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Correlations 


TgtC_Spd 

NEW  Spd  NEW  ImpRad  NEW_SnsrRng  NEW  TLEmu 

AC_Spd  AC 

LnchDist  C2 

DecTime 

TgtC_Spd 

1.0000 

-0.0004 

0.0003 

0.0000 

0.0000 

-0.0002 

0.0019 

"-0.0001 

NEW_Spd 

-0.0004 

1.0000 

0.0018 

0.0017 

0.0018 

-0.0001 

0.0005 

-0.0006 

NEWImpRad 

0.0003 

0.0018 

1.0000 

-0.0002 

0.0005 

0.0002 

0.0005 

-0.0003 

NEW_SnsrRng 

0.0000 

0.0017 

-0.0002 

1.0000 

0.0000 

0.0013 

-0.0003 

0.0005 

NEWTLEmu 

0.0000 

0.0018 

0.0005 

0.0000 

1.0000 

0.0009 

0.0011 

0.0005 

AC_Spd 

-0.0002 

-0.0001 

0.0002 

0.0013 

0.0009 

1.0000 

-0.0011 

-0.0009 

AC_LnchDist 

0.0019 

0.0005 

0.0005 

-0.0003 

0.0011 

-0.0011 

1.0000 

-0.0010 

C2  DecTime 

-0.0001 

-0.0006 

-0.0003 

0.0005 

0.0005 

-0.0009 

-0.0010 

1.0000 

Table  3.  Pairwise  Correlation  Matrix  Indicating  Minimal  Correlation  Among 

the  Columns  of  the  Design. 


The  NOLH  approach  used  in  this  research  created  257  unique  design  points,  or 
excursions.  Each  excursion  was  replicated  100  times,  for  a  total  of  25,700  simulated  tests 
for  each  scenario.  Chapter  IV  follows  with  a  discussion  of  the  results  and  an  explanation 
of  the  analytical  tools  used  to  process  the  simulation  output  data. 
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IV.  RESULTS  -  DATA  ANALYSIS 


The  purpose  of  computing  is  insight,  not  numbers. 


-Richard  Hamming 

This  chapter  covers  the  analysis  conducted  to  address  the  research  questions.  It  is 
important  to  note  that  the  intent  of  the  design  and  analysis  approach  in  this  thesis  was  to 
stress  the  system-of-systems  concept  in  ways  that  may  not  have  been  intended  by  the 
architects  of  specific  entities  within  the  construct.  Specifically,  while  every  attempt  was 
made  by  the  author  to  represent  realistic  characteristics  of  a  Network  Enabled  Weapon 
(NEW),  the  overall  objective  was  not  to  prove  or  demonstrate  a  certain  level  of 
capability.  Rather,  the  intent  was  to  gain  insight  into  system  performance  from  the 
standpoint  of  which  factors  (and  at  what  settings)  most  significantly  affects  overall 
system  behavior.  Review  of  the  results  and  analysis  from  this  research  must  be  kept  in 
context  with  the  description  of  the  model  and  the  scenarios,  as  previously  discussed. 
Extrapolation  of  the  results  beyond  the  particular  parameter  settings  used  in  the  designs  is 
discouraged — investigation  into  that  realm  remains  a  topic  for  future  study. 

A.  STATISTICAL  SETUP 

Analysis  of  the  model  output  was  conducted  with  JMP  Statistical  Discovery 
Software,  Version  5.1.  JMP  is  a  division  of  SAS,  a  leader  in  business  intelligence  and 
analysis,  and  is  an  accepted  standard  worldwide  in  the  research,  educational,  and  business 
fields.39  This  section  describes  the  procedures  used  to  extract  the  model  output  into 
useful  files  for  analysis.  Also  covered  in  this  section  is  a  general  introduction  to  the 
statistical  methods  used  to  interpret  the  results. 


39  See  http://www.jmpin.com.;  last  accessed  in  May  2007. 
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1. 


Procedures  for  Extracting  and  Summarizing  Model  Output 


Each  run  of  the  model  generated  an  output  data  file  that  contained  the  results  of 
the  Measures  of  Effectiveness  (MOEs)  discussed  in  Chapter  III.  In  order  to  equate  the 
output  to  the  corresponding  input  variables  for  a  specific  design  point,  a  Ruby40  script 
was  written  that  matched  the  design  matrix  to  the  output  and  created  a  single  comma 
separated  value  file.41  Each  row  of  the  output.csv  file  contains  an  organized  sequence  of 
values  for  the  input  parameters,  followed  by  the  specific  MOE  values  generated  by  the 
model  as  the  response.  In  the  case  of  an  experiment  consisting  of  257  design  points 
(DPs)  and  100  replications,  the  output.csv  file  would  contain  25,700  rows. 

Once  the  output  was  organized  into  an  acceptable  format,  the  data  was  imported 
into  JMP  for  further  analysis.  The  first  technique  involved  combining  the  data  into 
summary  statistics.  Since  the  same  inputs  were  used  in  each  replication  of  a  specific  DP, 
the  use  of  the  mean  value  of  the  MOE  across  all  replications  (in  this  case,  100)  serves  as 
a  “sufficient  statistic”  and  reduces  the  variance  within  a  set  of  inputs  while  preserving  all 
of  the  information  regarding  the  true  nature  of  the  response  42 

2.  Brief  Explanation  of  Stepwise  Regression  and  Partitioning  Trees 

The  two  main  tools  used  in  this  research  to  analyze  the  results  are 
Stepwise  Regression  and  Partitioning  Trees.  The  theory  behind  each  method  is  addressed 
here  briefly,  while  the  specific  application  of  the  approach  is  covered  in  more  detail  in  the 
following  sections,  as  results  are  presented. 

Regression  analysis  deals  with  the  stochastic  relationship  between  predictor 
variable(s)  (also  referred  to  as  explanatory  or  independent  variables),  and  at  least  one 
response  variable  of  interest.  The  procedure  is  useful  for  making  inferences  about  the 
effects  changes  to  the  inputs  have  on  the  output  and  provide  a  quantitative  measure  of  the 


40  See  http://ruby-lang.org/en.;  last  accessed  in  May  2007. 

41  Files  listed  with  a  .csv  extension  refer  to  comma  separated  value  format. 

42  V.K.  Rohatgi,  An  Introduction  to  Probability  Theory  and  Mathematical  Statistics,  John  Wiley  and 
Sons,  Inc.,  June  1976,  pp.  339-340. 
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extent  of  those  effects.43  In  JMP,  Stepwise  Regression  is  a  selection  within  the  model¬ 
fitting  platform  and  is  an  approach  to  selecting  a  subset  of  effects  for  a  regression  model. 
It  is  often  used  when  there  is  little  theory  to  guide  the  selection  of  terms  for  a  model,  but 
the  modeler  believes  the  approach  will  provide  a  good  fit.  The  approach  has  been  used 
practically,  however,  for  over  30  years  to  help  reduce  the  size  of  models  and  predict 
many  kinds  of  responses,  including  nonlinear  interactions.44 

Partitioning  Trees  provide  a  method  to  separate  the  predictor  variables  into  sets  of 
data  that  explain  significant  differences  in  the  response.  To  perform  each  “split”  of  the 
tree,  the  statistical  software  recursively  divides  the  factor  which  most  significantly 
separates  the  means  (in  the  case  of  continuous  variables)  by  examining  the  differences  in 
sums  of  squares.  The  method  is  often  very  useful,  especially  with  large  data  sets,  because 
there  are  no  prior  assumptions  required  regarding  the  underlying  distribution  of  the 
response  and,  as  will  be  shown,  analysis  of  the  results  is  very  straightforward.45 

B.  RESULTS  FOR  THE  ONE  TARGET  SCENARIO 

The  first  experiment  involved  the  examination  of  NEW  system  effectiveness  for 
the  case  of  one  mobile  target,  a  ground-based  weapon  controller,  and  an  aircraft  equipped 
with  one  NEW.  The  Design  of  Experiments  (DoE)  included  21  factors,  257  DPs,  and 
100  replications  of  each  DP.  The  complete  DoE  is  provided  in  Appendix  B. 

Figure  6  provides  the  initial  split  of  a  partition  tree  with  proportion  of  target  kills 
as  the  MOE.  The  top  block  indicates  an  overall  0.48  kill  rate  for  all  257  DPs.46 
Increases  in  the  proportion  of  target  kills  are  considered  an  improvement  in  the 
response — shown  as  branching  to  the  right  for  this  particular  tree.  As  the  figure 
indicates,  the  most  critical  factor  in  the  model  for  determining  kill  rate  is  the  target’s 
speed.  For  the  147  DPs  with  target  speed  set  less  than  13.19  m/s  (approximately  29.5 


43  Jay  L.  Devore,  Probability  and  Statistics  for  Engineering  and  the  Sciences,  6th  Edition,  2004, 
Thomson  Brooks/Cole  Publishers,  2004,  pp.  497-498. 

44  JMP  User’s  Guide,  Companion  Software  to  JMP  5.1,  2003. 

45  Ibid. 

46  The  number  of  data  points  in  each  block  will  be  shown  by  “Count”  on  the  partition  tree. 
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mph),  the  model  indicates  an  improvement  in  target  kills  to  74%,  opposed  to 
approximately  14%  for  the  other  110  cases.  Successive  splits  of  the  partitioning  tree 
occur  sequentially  for  the  parameter  explaining  the  most  variation  in  the  response. 


Figure  6.  First  Split  of  a  Partition  Tree  for  Target  Kills  as  the  MOE. 

The  results  shown  in  Figure  7  provide  a  more  complete  breakout  of  the  factors 
impacting  target  kills.  As  previously  mentioned,  interpretation  of  the  splits  for  this  MOE 
is  that  branching  to  the  right  indicates  an  improvement  in  the  response.  Each  box  should 
also  be  viewed  as  conditional  upon  the  parameter  value(s)  in  the  preceding  levels  of  the 
tree.  For  example,  the  block  in  the  bottom  right  comer  of  the  tree  contains  98  DPs  where 
the  JTAC  sensor  range  is  greater  than  or  equal  to  21 17  meters  and  greater  than  or  equal  to 
1150  meters  (which  is  trivial  in  this  case),  and  where  target  speed  is  less  than  13.19  m/s. 
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Figure  7.  First  Five  Splits  of  the  Partition  Tree  for  Target  Kills  as  the  MOE.47 

Two  important  points  are  revealed  in  Figure  7.  First,  if  the  operational  context 
includes  targets  traveling  greater  than  the  threshold  speed  of  29.5  mph,  then  the  model 
indicates  improvements  in  NEW  system  performance  may  still  be  achieved  if  the 
weapon’s  internal  sensor  range  is  greater  than  approximately  2,500  meters.  Secondly,  for 
targets  with  speeds  less  than  29.5  mph,  an  increase  in  the  JTAC’s  sensor  range  beyond 
approximately  2.1  kilometers  may  result  in  an  overall  80%  improvement  in  target  kills — 
shown  by  comparing  an  MOE  value  of  -0.90  in  the  lower  right  branch  to  the  0.48  value 
at  the  root  node.  The  apparent  significant  impact  of  the  time/distance  relationships 
exhibited  by  the  results  warrants  further  exploration. 

Regression  analysis  was  used  to  further  investigate  the  parameters  affecting  the 
simulation  model’s  response  of  target  kills.  The  first  approach  considered  the  main 
effects  only — meaning  only  the  factors  themselves,  without  consideration  of  second  order 
interactions  and  quadratics,  were  included  in  the  analysis.  Figure  8  shows  the 
Stepwise  Regression  control  panel  within  JMP.  The  figure  is  used  to  illustrate  the 
settings  chosen  for  the  regression  model.  The  user  selects  a  probability  to  enter  the 
model  and  a  probability  to  leave.  These  levels  indicate  the  statistical  significance  a 
potential  regressor  term  must  achieve  for  it  to  be  considered  in  the  model  in  a  forward  or 


47  Successive  splits  on  the  same  parameter  is  often  an  indicator  of  a  nonlinear  effect. 
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backward  step,  respectively.  The  direction  can  either  be  specified  as  forward,  backward, 
or  mixed.  The  mixed  setting  was  used  to  cause  the  process  to  alternate  between  forward 
(adding  terms)  and  backward  (removing  terms)  steps. 


Response:  MeantC-Killsl 

T  Stepwise  Regression  Control 


Prob  to  Enter 

0.050 

Prob  to  Leave 

0.100 

Direction:  Mixed  ▼ 


('  Enter  All  ) 
Remove  All  1 


f  Go  f  Stop  f  Step  )  (  Make  Model  ) 


Figure  8.  Stepwise  Regression  Control  Panel  in  JMP. 


The  summary  of  fit  shown  in  Figure  9  illustrates  the  R-squared  (R2)  and  adjusted 
R  values  for  the  resulting  model.  These  are  statistical  measures  that  indicate  the 
proportion  of  variation  in  the  response  attributable  to  terms  in  the  model,  as  opposed  to 
random  error.  An  R  value  of  1.0  would  indicate  the  model  perfectly  fit  the  data. 


Summary  of  Fit 

RSquare  0.582  328 

RSquare  Adj  0.574007 

Root  Mean  Square  Error  0.298896 

Mean  of  Response  0.483074 

Obse  rvat  i  o n  s  (o  r  Su  m  Wgts)  257 


Figure  9.  Summary  of  Fit  Resulting  from  Stepwise  Regression  for  Main 
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The  adjusted  R  value  is  especially  useful  in  stepwise  procedures  because  it 
provides  a  better  comparison  of  “fit”  across  models  by  using  the  degrees  of  freedom  in  its 
computation.48  The  adjusted  R2  value  shown  in  Figure  9  of  0.57  will  be  considered  a 
baseline  against  which  to  judge  other  regression  models  for  this  scenario. 

After  conducting  the  stepwise  reduction  procedure,  the  terms  with  a  statistically 
significant  impact  on  the  regression  model  are  shown  in  Figure  10.  The  results  are 
consistent  with  the  findings  for  the  partition  tree — namely  target  speed  and  JTAC  sensor 
range  contributing  the  most  to  the  response.  The  column  labeled  “Scaled  Estimate” 
provides  the  scaled  regression  coefficients  that  could  be  used  to  calculate  a  value  for  the 
expected  proportion  of  target  kills  for  a  given  set  of  input  variables.  The  scaling  provides 
a  meaningful  way  to  compare  coefficients  whose  corresponding  factors  may  have 
different  units.  A  clarifying  example  is  provided  in  Appendix  C.  Calculations  performed 
in  this  vein  are  only  valid  for  parameter  values  within  the  range  of  factor  settings  used  to 
generate  the  original  set  of  data.  The  fact  that  the  coefficient  for  target  speed  has  a 
negative  value  indicates  that  increases  in  speed  reduce  the  calculated  value  of  the 
response,  which  is  in  line  with  intuition  and  previous  findings. 


Scaled  Estimates 


Continuous  factors  centered  by  mean,  scaled  by  range/2 


Term 

Intercept 

TgtCSpd 

NEWImpRad 

NEWPdC 

ACSpd 

JTACSnsrRng 


Scaled  Estimate 

0.4830739 

-0.497853 

0.0654849 

0.0697697 

0.0808914 

0.3137775 


Std  Error 

0.018645 

0.032169 

0.032162 

0.032091 

0.032169 

0.032169 


t  Ratio 

Prob>|t| 

25.91 

<.0001 

-15.48 

<.0001 

2.04 

0.0428 

2.17 

0.0306 

2.51 

0.0125 

9.75 

<.0001 

Figure  10.  Scaled  Estimates  of  the  Regression  Coefficients  in  the  Main  Effects  Only 
Stepwise  Reduced  Statistical  Model  for  Target  Kills  as  the  MOE. 


48  Jay  L.  Devore,  Probability  and  Statistics  for  Engineering  and  the  Sciences,  6th  Edition,  2004, 
Thomson  Brooks/Cole  Publishers,  2004,  p.  580. 
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Running  the  stepwise  reduction  on  all  two-way  interactions  and  2nd  order 
polynomial  effects  provides  the  next  regression  model.  Figure  11  displays  the  scaled 
estimates  of  the  coefficients.  For  this  figure,  the  terms  have  been  sorted  based  on  the 
value  of  the  scaled  estimate.  This  selection  of  terms  produced  an  adjusted  R2  value 
of  0.63,  which  indicates  a  modest  improvement  in  the  explanatory  power  of  the  statistical 
model,  compared  to  the  model  with  only  main  effects  considered.  This  model  does, 
however,  provide  insight  into  the  interactions  between  the  parameters. 


|  Scaled  Estimates 

Continuous  factors  centered  by  mean,  scaled  by  range/2 

Term 

Scaled  Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

0.4665227 

■ 

0.039241 

11.89 

<0001 

JTACSnsrRng 

0.3138202 

0.029916 

10.49 

<0001 

(JTAC  _TLEmu-55.0039)*(JTAC  _TLEmu-55.0039) 

0.1676852 

0.05929 

2.83 

0.0051 

( AC_LnchIntrvl-2 0.0016)*  ( AC_LnchIntrvl-2 0.0016) 

0.1306483 

0.060957 

2.14 

0.0331 

(T  gt_C_Spd- 12.0012)*  (NE  W_SnsrRng-2 000 . 03 ) 

0.1281817 

0.055357 

2.32 

0.0214 

(NEW_PdC-0.92502)*(C2_DecTime- 105.001) 

0.1059225 

0.054574 

1.94 

0.0534 

AC_Spd 

0.0803891 

■ 

0.029918 

2.69 

0.0077 

NEW_PdC 

0.0716894 

I 

0.02985 

2.40 

0.0171 

NEW_ImpRad 

0.0634571 

I 

0.02993 

2.12 

0.0350 

NEW_SnsrRng 

0.0574092 

1 

0.029921 

1.92 

0.0562 

ACLnchlntrvl 

0.0424873 

□ 

0.02993 

1.42 

0.1570 

JTAC  TLEmu 

0.0418568 

n 

0.029924 

1.40 

0.1632 

C2_DecTime 

-0.038308 

f 

0.029928 

-1.28 

0.2018 

WCN_Rel 

-0.054672 

J] 

0.028912 

-1.89 

0.0598 

( JT  AC_SnsrRng-2 750.01)*  ( JT  AC_SnsrRng-2 750.01) 

-0.250201 

0.061826 

-4.05 

<0001 

Tgt_C_Spd 

-0.4973  | _ 

0.029918 

-16.62 

<0001 

Figure  1 1 .  Scaled  Estimates  of  the  Regression  Coefficients  for  the  Stepwise  Reduced 
Statistical  Model  with  all  Second  Order  Terms  Considered. 


It  should  also  be  noted  at  this  point  that  several  of  the  terms  listed  do  not  appear 
to  be  statistically  significant  at  the  .05  level — indicated  by  a  “Prob  >  |t|”  value  greater 
than  0.05.  Specifically,  the  significance  values  of  WCN  Reliability,  C2  Decision  Time, 
and  JTAC  TLEmu  fall  well  outside  of  this  range.  This  effect  is  a  result  of  statistical 
precedence,  which  indicates  terms  considered  significant  in  higher  order  interactions 
should  also  be  included  in  the  model  as  main  effects,  regardless  of  their 
individual  significance. 

Figure  12  shows  the  interaction  profiles  for  three  of  the  terms  in  the  regression 
model.  The  vertical  axis  along  the  left  side  indicates  the  value  of  the  response  variable, 
in  this  case  the  proportion  of  target  kills.  The  scale  along  the  bottom  indicates  the  range 
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of  settings  for  the  term  listed  in  the  respective  column.  The  numbers  on  the  interior  of 
the  plot  correspond  with  the  low  and  high  settings  for  the  term  listed  in  that  particular 
row.  For  example,  the  plot  in  the  first  row  and  second  column  displays  the  interaction 
between  target  speed  and  NEW  sensor  range.  The  values  of  4  and  20  are  the  low  and 
high  settings  for  speed,  respectively.  The  lines  being  nonparallel  indicate  an  interaction 
between  the  terms  is  present  in  the  response.  Specifically,  when  the  target’s  speed  is  low, 
the  response  is  minimally  affected  by  increasing  the  NEW’s  sensor  range.  If,  however, 
the  target  speed  is  high,  increases  in  the  NEW’s  sensor  range  have  a  significant  impact  on 
the  proportion  of  target  kills.  This  finding  is  consistent  with  the  left  branch  of  the 
partitioning  tree  shown  previously  in  Figure  7.  Also  of  interest  is  the  large  gap  between 
the  lines  on  the  top  row.  This  is  indicative  of  the  individual  effect  target  speed  has  on  the 
response. 


Figure  12.  Plot  of  Selected  Interaction  Terms  in  a  Regression  Model  for  Target  Kills 
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As  shown  by  the  distribution  in  Figure  13,  a  large  number  of  DPs  resulted  in  zero 
target  kills.  This  is  of  particular  interest  considering  the  data  used  for  the  analysis  is 
summary  data — meaning  all  100  replications  of  the  DP  would  have  to  result  in  missing 
the  target  for  it  to  register  as  a  proportion  equal  to  zero.  As  may  be  expected  from 
previous  results,  the  distributions  for  JTAC  sensor  range  and  target  speed  shown  in 
Figure  14  further  illuminate  their  effect  on  target  kills.  The  figure  is  generated  from  only 
the  119  DPs  with  a  mean  response  of  zero  and  illustrates  a  significant  portion  of  the  data 
contains  low  sensor  ranges  and  high  speeds.  It  appears  obvious  these  two  terms  are 
combining  to  create  a  threshold  where  the  simulation  model  indicates  the  system’s 
capability  to  identify,  engage,  and  prosecute  the  target  is  unwieldy.  A  contour  plot  is 
often  a  useful  tool  to  explore  this  possibility. 


Figure  13.  Distribution  of  Target  Kills  for  all  257  Design  Points  in  the 

Single  Target  Scenario. 
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Figure  14.  Distributions  of  Target  Speed  and  JTAC  Sensor  Range  for  119  Design 

Points  Where  No  Target  Kills  Occur. 

Figure  15  highlights  the  regions  of  the  response  where  the  combination  of  JTAC 
sensor  range  and  target  speed  result  in  the  gradations  shown  in  the  legend.  The  plot  not 
only  illustrates  the  combined  effect  of  the  two  parameters,  but  also  indicates  a  highly 
nonlinear  aspect — as  shown  by  the  superimposed  dotted  line. 
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Contour  Plot  for  Mean(C-Kills) 
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Figure  15.  Contour  Plot  Highlighting  the  Proportion  of  Target  Kills  as  a  Function  of 
the  Combination  Between  JTAC  Sensor  Range  and  Target  Speed. 

In  order  to  better  understand  the  combined  effect  of  JTAC  sensor  range  and  target 
speed,  and  to  more  fully  explore  other  potentially  influencing  parameters,  a  new  variable 
was  created  by  dividing  sensor  range  by  speed.  The  resulting  variable  has  units  of  time 
(specifically  seconds)  and  helps  reveal  previously  undiscovered  aspects  of  the  results.  In 
a  physical  sense,  the  variable  represents  the  time  it  would  take  a  target  moving  in  a 
straight  line  directly  at  the  JTAC  to  pass  completely  through  its  sensor  range.  To  avoid 
introducing  correlation  effects  into  the  statistical  models,  analysis  using  this  new  variable 
was  accomplished  with  the  original  variables  for  JTAC  sensor  range  and  target 
speed  excluded. 

The  partitioning  tree  shown  in  Figure  16  includes  the  new  variable  and  reveals 

insight  into  the  effect  of  C2  decision  time  and  launch  aircraft  speed.  Within  the  context 

of  factor  ranges  used  in  this  scenario,  the  first  two  splits  indicate  threshold  values  for  a 

notional  time  variable  given  by  dividing  the  range  and  speed  variables  as  discussed.  The 

results  indicate  a  time  value  exceeding  305  seconds  produces  favorable  system 

performance — a  proportion  of  target  kills  near  90%.  If  the  time  value  falls  between 

197  and  305  seconds,  then  the  simulation  model  suggests  C2  decision  time  and  aircraft 

speed  play  a  critical  role  in  overall  system  performance.  Specifically,  when  the  decision 
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to  initiate  an  attack  sortie  can  be  made  in  less  than  two  minutes  and  the  attack  aircraft  can 
fly  to  its  launch  standoff  distance  faster  than  180  m/s  (approximately  400  mph),  then  the 
estimate  for  proportion  of  kills  increases  from  55%  to  nearly  85%. 


|  All  Rows 

Count  257 

Mean  0.4830739 
StdDev  0.4579518 


Figure  16.  Partitioning  Tree  for  Proportion  of  Target  Kills  with  JTAC  Sensor  Range 
and  Target  Speed  Combined  Into  One  Variable. 


C.  RESULTS  FOR  THE  TWO-TARGET  SCENARIO 

The  second  experiment  involved  the  examination  of  NEW  system  effectiveness 
for  the  case  of  one  mobile  target,  one  dismounted  high-value  target  (HVT),  a 
ground-based  weapon  controller,  and  an  aircraft  equipped  with  two  NEWs.  The  DoE 
included  27  factors,  257  design  points  (DP),  and  100  replications  of  each  DP. 

Results  for  system  effectiveness  against  the  mobile  vehicle  are  very  similar  to  the 
one  target  case,  with  strong  influence  of  the  outcome  based  on  JTAC  sensor  range  and 
target  speed.  For  the  HVT,  though,  the  factors  most  influential  on  the  proportion  of  kills 
are  characteristics  of  the  weapon  itself. 
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Figure  17  illustrates  the  potential  effect  of  NEW  impact  radius  on  a  target  moving 
at  speeds  characteristic  of  a  dismounted  patrol.  The  HVT  speed  for  this  experiment 
ranged  from  0.5  to  4  m/s — the  difference  between  sauntering  around  and  running  just 
under  seven  minute  miles.  Overall,  the  simulation  results  show  an  88%  proportion  of 
HVT  kills.  For  NEW  impact  radii  exceeding  5.4  meters  and  probability  of  kill  (Pk) 
greater  than  0.92,  the  kill  level  approaches  95%.  As  a  reminder,  for  this  research,  Pk 
indicates  the  likelihood  of  successful  detonation  upon  impact.  It  can  also  be  thought  of  as 
a  conditional  probability,  meaning  the  probability  a  target  is  destroyed  given  its  location 
is  within  the  blast  radius.  On  the  left  branch  of  the  tree,  the  results  show  the  precision  of 
the  NEW  sensor  (i.e.,  TLE)  improves  system  performance  if  the  blast  radius  is  less  than 
5.4  meters.  The  same  explanation  applies  to  the  DPs  for  the  larger  impact  radius,  in  cases 
where  the  Pk  value  is  less  than  0.92. 


Figure  17.  Partitioning  Tree  for  Proportion  of  HVT  Kills. 

The  output  generated  for  this  research  did  not  support  an  indication  into  whether  a 
specific  kill  resulted  from  a  redirected  NEW  or  one  launched  specifically  designated  for 
the  HVT.  The  next  section,  however,  reveals  insight  regarding  the  effects  of  having  only 
one  NEW  available  for  the  attack  sortie. 
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D.  ADDITIONAL  FINDINGS 


1.  Results  for  Two  Targets  When  Only  One  NEW  is  Available 

The  results  in  this  section  were  obtained  using  the  same  DoE  as  in  the  previous 
two-target  case.  The  difference  here  is  the  fact  that  only  one  NEW  was  available  on  the 
attack  aircraft. 

Figure  1 8  portrays  the  results  for  proportion  of  HVT  kills.  As  previously  noted, 
the  intent  of  this  analysis  approach  is  to  gain  insight  into  which  factors  most  significantly 
affect  the  response  variable.  Therefore,  the  reader  is  cautioned  to  not  place  undue 
influence  on  the  value  of  the  MOE  itself,  but  instead  to  consider  the  relative  changes  in 
the  response  based  on  the  specific  value  of  the  parameter  under  examination. 


Figure  18.  Partitioning  Tree  for  Proportion  of  HVT  Kills  When 
Only  One  NEW  is  Available 

Unlike  results  for  the  one  target  case,  decreases  in  JTAC  sensor  range  tend  to 
improve  the  value  of  the  response.  This  is  likely  the  influence  of  early  detection  in  the 
simulation  of  the  mobile  vehicle  target.  If  the  JTAC’s  sensors  pick  up  the  vehicle  too 
soon,  then  the  attack  sortie  is  initiated  and  the  resulting  early  launch  of  the  NEW  limits 
the  ability  of  the  weapon  to  respond  to  a  request  for  redirect  to  the  HVT.  In  fact,  in  some 
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cases,  the  NEW  is  actually  flying  in  autonomous  seeker  mode  by  the  time  the  priority 
target  appears,  and  therefore  nonresponsive  to  communications  through  the  WCN.  This 
logic  is  further  supported  by  the  results  in  Figure  18  for  larger  JTAC  sensor  ranges 
(i.e.,  left  initial  branching).  Improvements  in  the  subsequent  split  indicate  a  drastic 
difference  in  the  MOE  if  the  vehicle  target  is  moving  slow — thereby  increasing  the  time 
before  initial  detection.  As  a  final  point  on  this  figure,  observe  that  even  when 
considering  the  DPs  where  JTAC  sensor  range  and  vehicle  target  speed  are  working 
against  HVT  success,  the  priority  target  may  still  appear  in  time  to  initiate  a  NEW 
redirect  if  it  materializes  in  the  scenario  less  than  350  seconds  after  the  start. 
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V.  CONCLUSIONS 


A.  ANALYSIS  SUMMARY 

This  research  considered  Network  Enable  Weapon  (NEW)  scenarios  involving  a 
ground-based  Joint  Terminal  Attack  Controller  (JTAC),  a  single  launch  aircraft,  and  two 
target  types.  The  targets  were  either  characteristic  of  a  mobile  tracked  vehicle  or  a 
dismounted  patrol.  The  dismounted  patrol  (when  present  in  the  scenario)  represented  a 
high-value  target  (HVT)  and,  therefore,  was  considered  a  higher  priority.  The  simulation 
model  contained  six  agent  types  and  allowed  for  the  exploration  of  effects  and 
interactions  for  up  to  27  parameters,  which,  of  course,  would  be  very  difficult  to 
accomplish  in  a  live  test. 

For  the  scenario  involving  one  mobile  target,  the  results  indicate  a  significant 
nonlinear  interaction  exists  between  the  speed  of  the  target  and  the  sensor  range  of  the 
JTAC.  Specifically,  for  targets  moving  less  than  approximately  30  mph  and  a  JTAC 
sensor  range  exceeding  two  kilometers,  the  model  indicates  an  80%  improvement  in  the 
system’s  ability  to  identify,  track,  engage,  and  kill  the  target.  Within  the  ranges  of 
parameter  settings  used  in  the  design,  these  findings  hold  regardless  of  the  other  factor 
settings.  JTAC  sensor  range  and  target  speed  were  combined  to  create  a  parameter  that 
notionally  represents  the  time  a  target  is  observable  by  the  ground  controller.  This 
approach  revealed  insight  into  the  effect  of  Command  and  Control  (C2)  decision  time  and 
the  aircraft’s  speed.  When  the  two  factors  combine  to  provide  moderate  target 
availability,  system  performance  increases  dramatically  if  the  amount  of  time  to  respond 
to  the  Close  Air  Support  (CAS)  request  is  kept  below  two  minutes  and  the  strike  aircraft 
proceeds  to  the  weapon  launch  point  at  a  speed  greater  than  400  mph. 

In  scenarios  involving  the  vehicle  target  and  the  HVT,  the  results  vary  depending 

on  the  weapons  load  of  the  strike  aircraft.  For  an  aircraft  carrying  two  NEWs,  the  ability 

to  successfully  engage  the  HVT  is  highly  dependent  on  the  weapons’  impact  radius  and 

accuracy.  When  only  one  weapon  is  available,  the  ability  to  attack  the  HVT  is 

significantly  influenced  by  the  timing  of  the  scenario.  For  all  designs  used  in  this 
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research,  the  vehicle  target  appears  10  seconds  after  the  start  of  the  simulation  run.  This 
means  any  attempts  to  prosecute  a  secondary  target  with  only  one  weapon  are  likely  to 
result  from  a  redirected  NEW.  If  the  HYT  appears  too  late  in  the  scenario,  or  if  the  JTAC 
sensor  range  and  vehicle  target  speed  combine  in  a  manner  that  results  in  an  early  CAS 
request,  then  the  model  indicates  the  system  is  unable  to  respond  to  a  NEW  redirect  to 
the  HVT. 

B.  KEY  INSIGHTS  SUPPORTING  USE  OF  AGENT-BASED  MODELING 

(ABM)  IN  TEST  PLANNING 

The  overall  objective  of  this  research  was  to  evaluate  the  use  of  ABM  as  a  tool  for 
enhancing  joint  test  and  evaluation.  As  a  proof  of  concept,  this  thesis  indicates  ABM, 
combined  with  efficient  experimental  design,  provides  relatively  quick  insight  into 
system-of-systems  performance  for  a  NEW  construct.  The  simulation  model  can  easily 
be  tailored  for  specific  scenarios.  When  paired  with  a  variety  of  designs,  a  myriad  of 
factor  settings  can  quickly  be  explored  in  order  to  gain  insight  into  specific  areas 
of  interest. 

Conducting  actual  live  tests  for  a  NEW  scenario  will  involve  months,  if  not  years, 
of  planning,  and  cost  several  millions  of  dollars.49  Using  ABM  techniques  before 
conducting  such  events  to  understand  potential  interactions  and  system  capabilities  is 
well  worth  the  investment.  When  properly  combined  with  the  right  simulation  tools,  the 
concepts  of  efficient  experimental  design  allow  for  an  investigation  of  system 
performance  across  a  broad  spectrum  of  parameters  and  factor  settings.  Considering  the 
broad  uncertainties  and  complex  interactions  typical  of  military  conflict,  ABM  offers  a 
worthwhile  analytical  approach  to  explore  elements  of  warfare  that  might  otherwise  be 
too  costly  or  time  intensive  to  study  by  other  means. 


49  Based  on  estimates  provided  by  JTEM  for  an  August  2007  planned  test  event. 
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C.  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

This  research  is  the  first  known  attempt  to  investigate  the  application  of  ABM  and 
efficient  experimental  design  in  support  of  JTEM  test  planning.  The  overall  concept 
appears  promising,  but  there  are  many  aspects  of  the  tools  and  techniques  used  in  this 
thesis  yet  to  be  fully  explored. 

1.  Experimental  Design 

The  designs  in  this  thesis  were  structured  to  quickly  explore  the  key  interactions 
in  a  NEW  system-of-systems  construct.  Many  of  the  initial  experiments  were  used  to 
error-check  the  model — making  it  critical  to  insure  the  simulations  could  be  conducted  in 
a  short  amount  of  computer  processing  time  on  one  machine.  Additionally,  once  the 
model  was  deemed  sufficiently  stable  for  conducting  analysis,  there  was  limited  time  to 
fully  explore  a  broader  range  of  parameters  and  parameter  settings.  Future  efforts  should 
consider  increasing  the  number  of  replications  for  each  design  point  beyond  the  100  used 
in  this  thesis.  Also,  there  are  advances  to  consider  in  the  application  of  nearly 
orthogonal,  space-filling  techniques,  offering  an  increase  in  the  number  of  factors  and 
factor  levels.  Each  of  these  approaches  will  likely  result  in  longer  computer  run  times  for 
the  experiment,  but  may  reveal  additional  insights  as  well. 

The  results  indicate  a  nominal  increase  in  the  predictive  power  of  the  regression 
formula  when  higher-order  effects  are  added  to  the  main  effects.  This  suggests  a 
sequential  screening  design  may  be  useful  in  developing  an  initial  estimate  of  which 
factors  to  consider  in  the  design.  The  space-filling  attributes  of  the  Nearly  Orthogonal 
Latin  Hypercube  (NOLH)  design,  however,  are  still  required  to  gain  insight  into  the 
complex  interactions  among  the  factors. 
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2. 


Simulation  Model 


The  simulation  model  is  essentially  a  prototype  constructed  to  facilitate  a  proof  of 
concept  investigation  of  ABM  applied  to  test  planning.  There  are  many  enhancements 
capable  of  improving  the  breadth  of  applications  to  the  NEW  construct,  a  few  of  which 
are  provided  here. 

•  Use  of  an  Unmanned  Aerial  Vehicle  (UAV)  to  identify  and  track 
targets  and  act  as  the  third-party  weapon  controller:  The  current 
research  was  limited  to  a  ground-based  controller.  The  extension  to  an 
aerial  based  controller  opens  the  possibility  of  increased  observation  time 
on  the  target  may  be  one  approach  to  addressing  the  issue  of  high-speed 
targets. 

•  Launch  aircraft  providing  In-Flight  Target  Updates  (IFTUs): 

Currently,  the  model  only  supports  in-flight  targeting  updates  from  the 
JTAC.  In  addition  to  adding  IFTU  capability  to  the  launch  aircraft, 
another  extension  would  involve  conditions  in  which  the  NEW  is 
designated  for  direct  attack  mode — meaning  the  weapon  goes  directly  into 
an  autonomous  mode  and  engages  the  target  with  its  own  sensors  upon 
launch. 

•  NEW  redirect  logic:  A  simplified  approach  was  used  to  determine 
whether  or  not  the  NEW  could  comply  with  a  request  to  redirect  to  a 
different  target.  A  more  thorough  physics-based  representation  is 
recommended  to  more  accurately  portray  actual  weapon  performance 
capabilities. 

•  Terrain  effects:  This  research  considered  operations  on  a  flat  surface. 
Adding  terrain  and/or  building  features  to  the  model  will  increase  analysis 
into  the  issues  of  line-of-sight  limitations. 

•  Target  awareness:  In  a  real-world  context,  targets  may  become  aware 
they  are  being  tracked  and  potentially  targeted  and  will  likely  alter  their 
behaviors  upon  such  recognition.  Advancements  to  the  model  should 
support  an  analysis  into  this  behavior. 

•  JTAC  FOV:  For  both  ground-based  and  UAV  controllers,  the  sensors  are 
typically  constrained  to  a  specific  field  of  view  (FOV).  This  research 
considered  a  360-degree  FOV,  which  is  a  somewhat  unrealistic  limitation. 

•  Randomized  factor  settings:  “Scripted”  settings  for  events  such  as  target 
initial  locations  and  waypoints  were  used  for  this  research.  Randomizing 
these  and  other  factor  settings  may  reveal  additional  insights  and  more 
accurately  represent  real-world  dynamics.  Other  factors  to  consider  for 
random  effects  include  target  pop-up  time,  number  of  targets,  target  speed, 
and  target  duration. 
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•  Fratricide  and  collateral  damage:  Operations  in  urban  or  constrained 
environments  are  prevalent  in  today’s  warfare.  Modeling  the  potential 
impact  to  unintended  targets,  whether  friendly  or  neutral,  may  provide  a 
useful  measure  of  system  effectiveness. 

•  Target  identification  to  kill  output  measure:  Including  time  from  target 
identification  to  kill  may  be  a  useful  additional  measure. 

3.  Scenarios  and  Factors 

There  are  obviously  numerous  scenarios  possible  for  both  real-world  NEW 
engagements  and  joint  test  events.  While  a  limited  set  were  explored  in  this  research,  the 
simulation  model  is  easily  adaptable  to  a  wide  range  of  possibilities. 

•  Time  Sensitive  Targeting:  Target  disappears  once  route  is  complete. 

•  Launch  aircraft  orbiting:  Rather  than  have  the  launch  aircraft  always 
initiate  from  a  set  point  in  space,  using  an  orbit  might  reveal  effects 
dictated  by  the  size,  type,  or  direction  of  the  orbit  pattern. 

•  Kill  criteria:  Different  target  sets  may  not  require  catastrophic 
destruction.  In  such  cases,  the  concept  of  obtaining  a  mobility  or 
firepower  kill  may  be  a  sufficient  objective  and  should  be  modeled 
accordingly. 

•  Multiple  targets:  As  in  real  life,  an  engagement  may  contain  multiple 
targets  and  targets  of  various  types  and  priorities.  An  additional  extension 
would  be  to  add  “legacy”  weapons  that  aren’t  network  enabled  to  the 
aircraft  load  and  investigate  the  effects  of  decisions  regarding  which 
weapon  type  to  use  against  a  specific  type  of  target  or  target  behavior. 

•  JTAC  Pd  toggle:  As  indicated  in  Chapter  III,  the  JTAC’s  set  detection 
probability  toggles  to  1.0  (meaning  certainty)  once  a  target  is  initially 
detected.  This  was  done  to  simulate  a  focused  attention  on  a  target,  but 
may  not  always  be  realistic.  Conducting  model  runs  in  which  the  Pd 
remains  at  the  setting  for  a  particular  design  point  (DP)  throughout  the 
engagement  may  reveal  different  results. 

4.  Process 

There  are  several  aspects  to  the  model  design  and  implementation  process  that 
could  be  improved  or  expanded.  Possibly  the  most  beneficial  expansion  would  be  to 
include  even  more  discussions  with  subject  matter  experts  (SMEs)  regarding  the  model’s 
functionality  and  capabilities.  The  “face  validity”  gained  by  exposing  the  model  to  SME 
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input  and  evaluation  would  increase  the  credibility  of  the  results  and  move  the  entire 
process  further  from  an  academic  endeavor  and  into  an  operationally  beneficial  product. 

From  a  usability  perspective,  the  process  of  initiating  model  runs  and  generating 
model  output  could  be  improved  to  enhance  the  portability  of  the  product.  As  it  currently 
stands,  the  model  requires  extensive  user  understanding  of  computer  programming 
interfaces  in  order  to  design  and  implement  a  specific  experiment.  Additionally,  the 
output  generates  a  massive  amount  of  data  files  that  must  be  processed  with  tools 
secondary  to  the  model  itself — generating  another  layer  of  analytical  complexity. 

It  is  anticipated  that  students  and  faculty  from  NPS  will  continue  working  with 
ITEM  to  expand  on  this  and  other  research  efforts  supporting  joint  test  planning.  The 
author  was  requested  to  comment  on  the  general  usefulness  of  the  current  model  in 
addressing  JTEM  requirements.  The  responses  are  provided  in  Appendix  D. 
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APPENDIX  A.  SUMMARY  DESCRIPTION  OF  PARAMETERS  IN 

THE  SIMULATION  MODEL 


This  table  provides  a  summary  description  of  the  factors  in  the  simulation  model. 
It  can  be  used  as  a  means  of  reference  when  viewing  figures  in  the  analysis  to  better 
understand  the  link  between  factor  name,  what  that  factor  represents,  and  the  units 
of  measure. 


Factor  Name 

Description 

Units 

JTAC  Spd 

JTAC  (i.e.,  third-party  weapon  controller)  Speed 

meters/sec 

JTACSnsrRng 

JTAC  Sensor  Range 

meters 

JTAC  PdA 

JTAC  Prob  Detection  for  HVT 

n/a 

JTAC  PdC 

JTAC  Prob  Detect  for  Mobile  Vehicle  Target 

n/a 

JTAC  MTDF 

JTAC  Multi-Target  Degradation  Factor 

n/a 

JTAC  TLEmu 

JTAC  Mean  Target  Location  Error 

meters 

JTACTLEsigma 

JTAC  Target  Location  Error  Standard  Deviation 

meters 

JTAC  IFTU  Intrvl 

JTAC  In-flight  Target  Update  Interval 

seconds 

NEW  Spd 

NEW  Speed 

meters/sec 

NEW  ImpRad 

NEW  Impact  Radius 

meters 

NEWSnsrRng 

NEW  Sensor  Range 

meters 

NEW  TLEmu 

NEW  Mean  Target  Location  Error 

meters 

NEW  TLEsigma 

NEW  Target  Location  Error  Standard  Deviation 

meters 

NEWPdA 

NEW  Prob  Detection  for  HVT 

n/a 

NEWPdC 

NEW  Prob  Detection  for  Mobile  Vehicle  Target 

n/a 

NEWPkA 

NEW  Prob  Kill  for  HVT 

n/a 

NEWPkC 

NEW  Prob  Kill  for  Mobile  Vehicle  Target 

n/a 

TgtASpd 

HVT  Speed 

meters/sec 

TgtAPUT 

HVT  Pop-up  Time 

seconds 

TgtCSpd 

Target  C  Speed 

meters/sec 

TgtCPUT 

Target  C  Pop-up  Time 

seconds 

ACSpd 

Aircraft  Speed 

meters/sec 

ACLnchDist 

Aircraft  Stand-off  Launch  Distance 

meters 

ACUpdtReq 

Aircraft  Requirement  for  Recency  of  Target  Info 

seconds 

ACLnchAtmpts 

Aircraft  Number  of  Launch  Attempts 

n/a 

ACLnchlntrvl 

Aircraft  Interval  Between  Launch  Attempts 

seconds 

WCNRel 

Weapon  Control  Network  Reliability 

n/a 

WCNLat 

Weapon  Control  Network  Latency 

n/a 

C2_DecTime 

Command  and  Control  Decision  Time 

seconds 
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APPENDIX  B.  EXPERIMENTAL  DESIGN  FOR  THE  SINGLE 

TARGET  SCENARIO 


Low 

4 

140 

3 

1000 

0 

0 

0.85 

0.9 

154 

18500 

10 

High 

20 

200 

30 

3000 

5 

2 

1 

1 

254 

74000 

45 

Decimals 

2 

0 

1 

0 

2 

2 

2 

2 

1 

0 

0 

Design  Pt  # 

Tgt  C  Spd 

NEW  Spd 

NEW  ImpRad 

NEW  SnsrRng 

NEW  TLEmu 

NEW  TLEsigma 

NEW  PdC 

NEW  PkC 

AC  Spd 

AC  LnchDist 

AC  UpdtReq 

1 

13.5 

178 

27.3 

3000 

4.38 

0.38 

0.89 

0.91 

172.4 

37361 

24 

2 

17.44 

183 

17.3 

2945 

4.22 

1.09 

0.94 

1 

242.7 

46900 

11 

3 

5.88 

198 

24.9 

2883 

4.45 

1.29 

0.88 

0.91 

168.5 

70531 

39 

4 

10.38 

184 

23.1 

2758 

4.75 

0.3 

0.99 

1 

249.3 

54271 

30 

5 

13.81 

196 

24.4 

2508 

3.44 

0.13 

0.86 

0.93 

161.8 

46467 

16 

6 

19.31 

192 

27.5 

2469 

4.77 

1.45 

0.98 

0.98 

224.7 

69230 

25 

7 

11.06 

171 

17 

2531 

2.93 

1.34 

0.85 

0.93 

186 

26521 

42 

8 

9.88 

196 

23 

2398 

3.4 

0.97 

0.92 

0.98 

219.2 

42131 

42 

9 

18.88 

163 

24.2 

2477 

4.9 

0.35 

0.99 

0.95 

197 

28689 

33 

10 

17.81 

143 

19.5 

2133 

2.75 

1.91 

0.91 

0.98 

248.9 

25871 

30 

11 

11.75 

166 

16.6 

2953 

2.99 

1.8 

0.95 

0.92 

194.2 

54488 

14 

12 

5.19 

162 

21 

2539 

3.11 

0.98 

0.92 

1 

217.7 

58391 

25 

13 

14.06 

164 

24.5 

2289 

3.32 

0.74 

0.96 

0.9 

201.3 

63594 

33 

14 

15.5 

150 

24.6 

2977 

2.56 

1.43 

0.9 

0.97 

220 

51887 

40 

15 

4.63 

147 

17.2 

2523 

2.81 

1.48 

0.93 

0.9 

168.8 

24570 

21 

16 

8.31 

164 

22.5 

2875 

4.28 

0.21 

0.85 

0.98 

211.8 

28256 

23 

17 

16.75 

194 

13 

2281 

3.3 

0.31 

0.93 

0.91 

186.8 

36928 

22 

18 

16.81 

172 

10.6 

2703 

3.38 

1.55 

0.99 

0.96 

216.9 

44949 

16 

19 

11.56 

189 

4.1 

2742 

3.95 

1.35 

0.89 

0.92 

163.8 

56873 

28 

20 

8.44 

185 

10.3 

2555 

4.59 

0.93 

0.97 

0.99 

234.9 

61426 

40 

21 

16.56 

188 

4.9 

2969 

4.82 

0.75 

0.85 

0.91 

170.8 

69881 

20 

22 

13.75 

194 

6.7 

2734 

4.12 

1.53 

0.96 

0.99 

252.4 

45383 

20 

23 

11.94 

171 

16.1 

2563 

3.63 

1.38 

0.88 

0.92 

189.5 

34977 

44 

24 

9.31 

185 

5 

2633 

3.28 

0.95 

0.92 

0.97 

228.2 

32592 

34 

25 

16.69 

153 

13.4 

2070 

4.98 

0.14 

0.94 

0.93 

159.5 

41264 

37 

26 

18.5 

163 

4.2 

2367 

4.43 

1.52 

0.88 

0.95 

250.1 

35193 

28 

27 

11.69 

170 

14.9 

2211 

4.04 

1.44 

0.95 

0.93 

199.7 

49285 

25 

28 

8.13 

160 

12.9 

2391 

2.71 

0.95 

0.93 

0.96 

234.5 

72916 

18 

29 

18.38 

152 

14 

2672 

3.87 

0.98 

0.96 

0.92 

191.9 

65545 

38 

30 

12.5 

152 

7.3 

2500 

2.87 

1.8 

0.87 

0.98 

227.8 

54055 

24 

31 

7 

168 

6.2 

2344 

4.55 

1.42 

0.95 

0.94 

159.9 

25221 

16 

32 

8.06 

157 

13.8 

2078 

4.67 

0.42 

0.91 

0.96 

243.8 

30641 

12 

33 

17.38 

178 

30 

1203 

3.54 

0.66 

0.87 

0.96 

196.6 

26305 

16 

34 

16 

191 

29.3 

1938 

2.89 

1.81 

0.94 

0.92 

229 

39529 

18 

35 

9.25 

192 

28.4 

1375 

2.85 

1.16 

0.88 

0.98 

163 

49068 

40 

36 

11.38 

187 

26.7 

1508 

4.84 

0.86 

0.98 

0.92 

213 

48635 

36 

37 

12.56 

199 

23.4 

1414 

4.16 

0.84 

0.86 

0.99 

199.3 

44516 

12 

38 

14.94 

192 

22.8 

1188 

2.91 

1.15 

1 

0.92 

238.4 

49719 

15 

39 

10.31 

187 

23.7 

1961 

3.46 

1.51 

0.87 

0.98 

186.4 

29123 

40 

40 

8.88 

189 

21.9 

1516 

3.57 

0.94 

0.96 

0.95 

227.4 

24787 

44 

41 

19.75 

168 

22.9 

1430 

3.55 

0.33 

0.95 

0.96 

189.2 

24354 

32 

42 

17.88 

159 

18.3 

1781 

3.09 

1.95 

0.92 

0.92 

222.8 

27822 

35 

43 

7.5 

164 

29.4 

1992 

4.73 

1.82 

0.95 

0.97 

195.8 

62943 

21 

44 

6.94 

158 

23.8 

1664 

3.13 

0.52 

0.91 

0.95 

251.7 

62510 

29 

45 

14.25 

150 

20.4 

1406 

2.54 

0.39 

0.96 

0.98 

157.5 

58174 

26 

46 

17.63 

155 

29.7 

1398 

4.51 

1.6 

0.86 

0.93 

202 

68363 

31 

47 

6.06 

160 

23.6 

1945 

2.95 

1.72 

0.92 

0.97 

200.9 

21102 

17 

48 

7.56 

168 

28.3 

1555 

2.73 

0.41 

0.89 

0.95 

217.3 

20885 

13 

49 

14.31 

200 

12.7 

1742 

2.62 

0.02 

0.86 

0.97 

164.5 

33242 

13 

50 

19.81 

198 

7 

1563 

4.61 

1.2 

1 

0.93 

203.2 

18934 

12 

51 

7.81 

196 

6.5 

1078 

4.3 

1.77 

0.88 

0.97 

193.8 

53404 

42 

52 

5 

193 

9 

1539 

3.24 

0.32 

1 

0.9 

243.1 

53621 

45 

53 

15.81 

185 

3.4 

1141 

2.77 

0.81 

0.89 

0.99 

184.9 

73133 

16 

54 

13.06 

184 

6.6 

1273 

3.26 

1.47 

0.96 

0.9 

207.5 

56223 

14 

55 

4.38 

186 

8.9 

1969 

4.96 

1.59 

0.88 

0.99 

174.7 

25004 

32 

56 

7.69 

182 

8 

1148 

4.24 

0.07 

0.96 

0.94 

254 

37578 

29 

57 

16.06 

156 

15.6 

1773 

3.79 

0.5 

0.94 

0.99 

169.2 

26088 

35 

58 

15.75 

166 

11.5 

1086 

4.1 

1.45 

0.9 

0.94 

225.5 

22186 

45 

59 

7.75 

141 

13.7 

1883 

4.14 

1.41 

0.97 

0.96 

162.2 

73783 

25 

60 

8.81 

154 

11.2 

1734 

4.47 

0.38 

0.86 

0.94 

233.7 

74000 

21 

61 

20 

161 

7.4 

1813 

3.52 

0.05 

1 

0.98 

166.9 

59691 

41 

62 

19.56 

141 

9.8 

1320 

4.02 

1.98 

0.9 

0.94 

213.4 

66846 

33 

63 

4.94 

154 

11.9 

1234 

3.18 

1.84 

0.99 

0.96 

161.4 

33025 

10 

64 

5.94 

144 

15.4 

1797 

4.57 

0.16 

0.88 

0.91 

215.3 

41047 

22 

65 

15.25 

171 

22.1 

2750 

0 

0.3 

0.87 

0.92 

253.2 

20451 

26 

66 

16.88 

195 

18.6 

2688 

0.14 

1.27 

0.94 

0.99 

155.2 

23486 

18 

67 

9.81 

192 

18.4 

2781 

0.29 

1.71 

0.91 

0.91 

247.8 

53838 

44 

68 

5.38 

179 

29.2 

2898 

0.61 

0.72 

0.98 

0.99 

168.1 

56006 

34 

69 

16.13 

173 

25.5 

2375 

1.23 

0.37 

0.9 

0.91 

227 

65762 

18 

70 

17.13 

179 

18.7 

2906 

1.33 

1.52 

0.93 

0.97 

179.4 

56656 

10 

71 

6.75 

200 

21.7 

2172 

1.17 

1.83 

0.87 

0.94 

230.6 

21535 

35 

72 

5.69 

191 

22.3 

2359 

1.5 

0.34 

0.99 

0.96 

183.7 

32158 

28 

73 

12.88 

155 

22.2 

2961 

1.31 

0.69 

0.97 

0.95 

210.3 

33459 

42 

74 

14.44 

151 

19.7 

2102 

2.17 

1.04 

0.89 

1 

178.2 

30424 

41 

75 

4.13 

150 

28.5 

2195 

0.12 

1.73 

0.99 

0.94 

234.1 

64895 

19 

76 

6.44 

146 

19.9 

2242 

1.15 

0.04 

0.88 

0.96 

156 

54705 

23 

77 

12.38 

158 

16.7 

2328 

1.78 

0.11 

0.97 

0.95 

251.3 

60559 

43 

78 

18.75 

152 

27.4 

2023 

0.06 

1.08 

0.89 

0.97 

185.6 

72482 

32 

79 

6.25 

162 

18.9 

2125 

1.19 

1.39 

0.98 

0.91 

250.9 

35410 

17 

80 

9.63 

145 

17.8 

2711 

0.31 

0.26 

0.9 

0.95 

177 

48418 

19 

81 

12.13 

182 

15.9 

2320 

1.8 

0.09 

0.92 

0.93 

240.7 

42348 

12 

55 


Low 

4 

140 

3 

1000 

0 

0 

0.85 

0.9 

154 

18500 

10 

High 

20 

200 

30 

3000 

5 

2 

1 

1 

254 

74000 

45 

Decimals 

2 

0 

1 

0 

2 

2 

2 

2 

1 

0 

0 

Design  Pt  # 

Tgt  C  Spd 

NEW  Spd 

NEW  ImpRad 

NEW  SnsrRng 

NEW  TLEmu 

NEW  TLEsigma 

NEW  PdC 

NEW  PkC 

AC  Spd 

AC  LnchDist 

AC  UpdtReq 

82 

18.44 

175 

5.1 

2352 

0.74 

0.87 

0.97 

0.96 

184.1 

19801 

26 

83 

10.56 

174 

6.8 

2578 

0.64 

0.9 

0.87 

0.9 

232.5 

55355 

33 

84 

11.25 

198 

12.5 

2836 

1.11 

0.64 

0.94 

0.98 

170.4 

51670 

38 

85 

15.38 

190 

15 

2930 

0.08 

0.59 

0.92 

0.92 

241.5 

44299 

24 

86 

13.88 

175 

12.4 

2648 

0.66 

1.4 

0.97 

1 

154.4 

49502 

18 

87 

4.88 

181 

3.2 

2453 

1.09 

1.01 

0.9 

0.93 

238 

27172 

29 

88 

10 

183 

7.1 

2313 

0.92 

0.03 

0.96 

0.97 

175.1 

41480 

38 

89 

17.31 

157 

9.5 

2992 

2.32 

0.7 

0.94 

0.91 

218.1 

21752 

35 

90 

13.31 

163 

7.9 

2773 

1.58 

1.54 

0.9 

1 

198.5 

44732 

43 

91 

8.94 

143 

7.6 

2617 

1.97 

1.3 

0.94 

0.94 

220.8 

57957 

22 

92 

8.56 

163 

5.8 

2086 

1.52 

0.34 

0.87 

0.98 

193.1 

60125 

19 

93 

15.31 

170 

11 

2547 

0.82 

0.8 

0.99 

0.94 

225.9 

58607 

28 

94 

13.25 

146 

8.3 

2148 

1.25 

1.33 

0.93 

0.97 

165.7 

62727 

28 

95 

10.88 

165 

12.8 

2820 

1.64 

1.9 

0.93 

0.91 

207.9 

45166 

16 

96 

4.5 

167 

5.3 

2867 

2.3 

0.76 

0.91 

0.99 

188.4 

31725 

24 

97 

16.38 

180 

26.6 

1586 

0.51 

0.89 

0.91 

0.96 

209.1 

33676 

13 

98 

13.19 

181 

25.8 

1844 

2.34 

1.64 

0.95 

0.93 

167.7 

28473 

25 

99 

5.44 

187 

27 

1859 

0.94 

1.88 

0.87 

1 

236 

61209 

32 

100 

5.06 

195 

28.6 

1063 

1.27 

0.23 

0.97 

0.95 

181.7 

71182 

36 

101 

19.94 

198 

21.6 

1336 

1.04 

0.27 

0.87 

0.96 

244.6 

60992 

19 

102 

18.19 

189 

28.7 

1836 

0.47 

1.99 

1 

0.92 

177.8 

62293 

23 

103 

7.06 

184 

18.8 

1617 

2.4 

1.09 

0.89 

0.97 

232.1 

40180 

36 

104 

11.31 

179 

21.4 

1570 

1.29 

0.44 

0.95 

0.93 

203.6 

28039 

43 

105 

19.44 

140 

29.5 

1578 

1.07 

0.45 

0.99 

0.97 

209.9 

19150 

29 

106 

17.19 

147 

17.9 

1766 

1.95 

1.94 

0.85 

0.93 

171.6 

39963 

43 

107 

8.38 

151 

19.1 

1109 

2.48 

1.37 

0.94 

0.96 

220.4 

52104 

11 

108 

9.5 

167 

19.8 

1750 

1.66 

0.51 

0.89 

0.91 

205.2 

55789 

15 

109 

14.56 

154 

20.9 

1984 

1.02 

0.77 

0.93 

0.99 

231.3 

52754 

41 

110 

14.81 

166 

16.8 

1195 

1 

1.38 

0.88 

0.95 

172.8 

63160 

40 

111 

7.38 

145 

18.2 

1820 

2.36 

1.92 

0.98 

0.95 

225.1 

23053 

24 

112 

5.31 

144 

26.1 

1906 

1.39 

0.54 

0.86 

0.94 

202.4 

22836 

18 

113 

14.63 

193 

12.2 

1953 

1.86 

0.88 

0.88 

0.99 

226.7 

41914 

11 

114 

12.19 

191 

11.8 

1156 

1.41 

1.23 

0.99 

0.95 

192.3 

36061 

14 

115 

11 

193 

8.7 

1281 

0.2 

1.57 

0.87 

0.96 

229.4 

49936 

37 

116 

6.31 

197 

5.2 

1703 

1.35 

0.68 

0.99 

0.91 

158.3 

68580 

44 

117 

19.69 

181 

3.9 

1891 

0.35 

0.73 

0.86 

0.96 

236.8 

71832 

17 

118 

12.81 

197 

7.7 

1695 

0.68 

1.85 

0.96 

0.94 

160.6 

41697 

24 

119 

10.44 

175 

10.4 

1016 

2.42 

1.17 

0.9 

0.98 

247 

46684 

40 

120 

10.06 

181 

12.3 

1305 

0.37 

0.79 

0.94 

0.9 

201.7 

30857 

34 

121 

15 

141 

3.1 

1484 

1.93 

0 

0.94 

0.97 

241.9 

27389 

28 

122 

19.25 

167 

6.1 

1359 

0.21 

1.12 

0.87 

0.92 

184.5 

20234 

34 

123 

10.63 

164 

8.2 

1344 

2.21 

1.18 

0.95 

0.99 

214.5 

68797 

21 

124 

9.13 

163 

15.3 

1211 

1.84 

0.78 

0.92 

0.91 

197.4 

67063 

14 

125 

18 

160 

9.1 

1594 

2.03 

0.24 

0.93 

0.98 

231.7 

57740 

38 

126 

17.5 

169 

14.5 

1391 

0.8 

1.78 

0.9 

0.93 

191.5 

48852 

28 

127 

5.75 

166 

5.4 

1727 

0.59 

1.75 

0.98 

0.99 

212.6 

22402 

24 

128 

4.81 

149 

4.8 

1172 

1.99 

0.13 

0.92 

0.93 

162.6 

39313 

20 

129 

12 

170 

16.5 

2000 

2.5 

1 

0.93 

0.95 

204 

46250 

28 

130 

10.5 

162 

5.7 

1000 

0.63 

1.63 

0.96 

0.99 

235.6 

55139 

31 

131 

6.56 

157 

15.7 

1055 

0.78 

0.91 

0.91 

0.9 

165.3 

45600 

44 

132 

18.13 

142 

8.1 

1117 

0.55 

0.71 

0.97 

0.99 

239.5 

21969 

16 

133 

13.63 

156 

9.9 

1242 

0.25 

1.7 

0.86 

0.9 

158.7 

38229 

25 

134 

10.19 

144 

8.6 

1492 

1.56 

1.87 

0.99 

0.97 

246.2 

46033 

39 

135 

4.69 

148 

5.5 

1531 

0.23 

0.55 

0.87 

0.92 

183.3 

23270 

30 

136 

12.94 

169 

16 

1469 

2.07 

0.66 

1 

0.97 

222 

65979 

13 

137 

14.13 

144 

10 

1602 

1.6 

1.03 

0.93 

0.92 

188.8 

50369 

13 

138 

5.13 

177 

8.8 

1523 

0.1 

1.65 

0.86 

0.95 

211 

63811 

22 

139 

6.19 

197 

13.5 

1867 

2.25 

0.09 

0.94 

0.92 

159.1 

66629 

25 

140 

12.25 

174 

16.4 

1047 

2.01 

0.2 

0.9 

0.98 

213.8 

38012 

41 

141 

18.81 

178 

12 

1461 

1.89 

1.02 

0.93 

0.9 

190.3 

34109 

30 

142 

9.94 

176 

8.5 

1711 

1.68 

1.26 

0.89 

1 

206.7 

28906 

22 

143 

8.5 

190 

8.4 

1023 

2.44 

0.57 

0.95 

0.93 

188 

40613 

15 

144 

19.38 

193 

15.8 

1477 

2.19 

0.52 

0.92 

1 

239.2 

67930 

34 

145 

15.69 

176 

10.5 

1125 

0.72 

1.79 

1 

0.92 

196.2 

64244 

32 

146 

7.25 

146 

20 

1719 

1.7 

1.69 

0.92 

0.99 

221.2 

55572 

33 

147 

7.19 

168 

22.4 

1297 

1.62 

0.45 

0.86 

0.94 

191.1 

47551 

39 

148 

12.44 

151 

28.9 

1258 

1.05 

0.65 

0.96 

0.98 

244.2 

35627 

27 

149 

15.56 

155 

22.7 

1445 

0.41 

1.07 

0.88 

0.91 

173.1 

31074 

15 

150 

7.44 

152 

28.1 

1031 

0.18 

1.25 

1 

0.99 

237.2 

22619 

35 

151 

10.25 

146 

26.3 

1266 

0.88 

0.47 

0.89 

0.91 

155.6 

47117 

35 

152 

12.06 

169 

16.9 

1438 

1.37 

0.63 

0.97 

0.98 

218.5 

57523 

11 

153 

14.69 

155 

28 

1367 

1.72 

1.05 

0.93 

0.93 

179.8 

59908 

21 

154 

7.31 

187 

19.6 

1930 

0.02 

1.86 

0.91 

0.97 

248.5 

51236 

18 

155 

5.5 

177 

28.8 

1633 

0.57 

0.48 

0.97 

0.95 

157.9 

57307 

27 

156 

12.31 

170 

18.1 

1789 

0.96 

0.56 

0.9 

0.97 

208.3 

43215 

30 

157 

15.88 

180 

20.1 

1609 

2.29 

1.05 

0.92 

0.94 

173.5 

19584 

37 

158 

5.63 

188 

19 

1328 

1.13 

1.02 

0.89 

0.98 

216.1 

26955 

17 

159 

11.5 

188 

25.7 

1500 

2.13 

0.2 

0.98 

0.92 

180.2 

38445 

31 

160 

17 

172 

26.8 

1656 

0.45 

0.58 

0.9 

0.96 

248.1 

67279 

39 

161 

15.94 

183 

19.2 

1922 

0.33 

1.58 

0.94 

0.94 

164.2 

61859 

43 

162 

6.63 

162 

3 

2797 

1.46 

1.34 

0.98 

0.94 

211.4 

66195 

39 

163 

8 

149 

3.7 

2063 

2.11 

0.19 

0.91 

0.98 

179 

52971 

37 

164 

14.75 

148 

4.6 

2625 

2.15 

0.84 

0.97 

0.92 

245 

43432 

15 

165 

12.63 

153 

6.3 

2492 

0.16 

1.14 

0.87 

0.98 

195 

43865 

19 

166 

11.44 

141 

9.6 

2586 

0.84 

1.16 

0.99 

0.91 

208.7 

47984 

43 

167 

9.06 

148 

10.2 

2813 

2.09 

0.85 

0.85 

0.98 

169.6 

42781 

40 

168 

13.69 

153 

9.3 

2039 

1.54 

0.49 

0.98 

0.92 

221.6 

63377 

15 

169 

15.13 

151 

11.1 

2484 

1.43 

1.06 

0.89 

0.95 

180.6 

67713 

11 

57 


Low 

4 

140 

3 

1000 

0 

0 

0.85 

0.9 

154 

18500 

10 

High 

20 

200 

30 

3000 

5 

2 

1 

1 

254 

74000 

45 

Decimals 

2 

0 

1 

0 

2 

2 

2 

2 

1 

0 

0 

Design  Pt  # 

Tgt  C  Spd 

NEW  Spd 

NEW  ImpRad 

NEW  SnsrRng 

NEW  TLEmu 

NEW  TLEsigma 

NEW  PdC 

NEW  PkC 

AC  Spd 

AC  LnchDist 

AC  UpdtReq 

170 

4.25 

172 

10.1 

2570 

1.45 

1.67 

0.9 

0.94 

218.8 

68146 

23 

171 

6.13 

181 

14.7 

2219 

1.91 

0.05 

0.93 

0.98 

185.3 

64678 

20 

172 

16.5 

176 

3.6 

2008 

0.27 

0.18 

0.9 

0.93 

212.2 

29557 

34 

173 

17.06 

182 

9.2 

2336 

1.88 

1.48 

0.94 

0.95 

156.3 

29990 

26 

174 

9.75 

190 

12.6 

2594 

2.46 

1.61 

0.89 

0.92 

250.5 

34326 

29 

175 

6.38 

185 

3.3 

2602 

0.49 

0.4 

0.99 

0.98 

206 

24137 

24 

176 

17.94 

180 

9.4 

2055 

2.05 

0.28 

0.93 

0.93 

207.1 

71398 

38 

111 

16.44 

172 

4.7 

2445 

2.27 

1.59 

0.96 

0.95 

190.7 

71615 

42 

178 

9.69 

140 

20.3 

2258 

2.38 

1.98 

0.99 

0.93 

243.5 

59258 

42 

179 

4.19 

142 

26 

2438 

0.39 

0.8 

0.85 

0.97 

204.8 

73566 

43 

180 

16.19 

144 

26.5 

2922 

0.7 

0.23 

0.97 

0.93 

214.2 

39096 

13 

181 

19 

147 

24 

2461 

1.76 

1.68 

0.85 

1 

164.9 

38879 

10 

182 

8.19 

155 

29.6 

2859 

2.23 

1.19 

0.96 

0.91 

223.1 

19367 

39 

183 

10.94 

156 

26.4 

2727 

1.74 

0.53 

0.89 

1 

200.5 

36277 

41 

184 

19.63 

154 

24.1 

2031 

0.04 

0.41 

0.97 

0.91 

233.3 

67496 

23 

185 

16.31 

158 

25 

2852 

0.76 

1.93 

0.89 

0.96 

154 

54922 

26 

186 

7.94 

184 

17.4 

2227 

1.21 

1.5 

0.91 

0.91 

238.8 

66412 

20 

187 

8.25 

174 

21.5 

2914 

0.9 

0.55 

0.95 

0.96 

182.5 

70314 

10 

188 

16.25 

199 

19.3 

2117 

0.86 

0.59 

0.88 

0.94 

245.8 

18717 

30 

189 

15.19 

186 

21.8 

2266 

0.53 

1.62 

0.99 

0.96 

174.3 

18500 

34 

190 

4 

179 

25.6 

2188 

1.48 

1.95 

0.85 

0.92 

241.1 

32809 

14 

191 

4.44 

199 

23.3 

2680 

0.98 

0.02 

0.95 

0.96 

194.6 

25654 

22 

192 

19.06 

186 

21.1 

2766 

1.82 

0.16 

0.86 

0.94 

246.6 

59475 

45 

193 

18.06 

196 

17.6 

2203 

0.43 

1.84 

0.97 

0.99 

192.7 

51453 

33 

194 

8.75 

169 

10.9 

1250 

5 

1.7 

0.98 

0.98 

154.8 

72049 

29 

195 

7.13 

145 

14.4 

1313 

4.86 

0.73 

0.91 

0.91 

252.8 

69014 

37 

196 

14.19 

148 

14.6 

1219 

4.71 

0.29 

0.94 

0.99 

160.3 

38662 

11 

197 

18.63 

161 

3.8 

1102 

4.39 

1.28 

0.87 

0.91 

239.9 

36494 

21 

198 

7.88 

167 

7.5 

1625 

3.77 

1.63 

0.95 

0.99 

181 

26738 

37 

199 

6.88 

161 

14.3 

1094 

3.67 

0.48 

0.92 

0.93 

228.6 

35844 

45 

200 

17.25 

140 

11.3 

1828 

3.83 

0.17 

0.98 

0.96 

177.4 

70965 

20 

201 

18.31 

149 

10.7 

1641 

3.5 

1.66 

0.86 

0.94 

224.3 

60342 

27 

202 

11.13 

185 

10.8 

1039 

3.69 

1.31 

0.88 

0.95 

197.8 

59041 

13 

203 

9.56 

189 

13.3 

1898 

2.83 

0.96 

0.96 

0.9 

229.8 

62076 

14 

204 

19.88 

190 

4.5 

1805 

4.88 

0.27 

0.86 

0.96 

173.9 

27605 

36 

205 

17.56 

194 

13.1 

1758 

3.85 

1.96 

0.97 

0.94 

252 

37795 

32 

206 

11.63 

182 

16.3 

1672 

3.22 

1.89 

0.88 

0.95 

156.7 

31941 

12 

207 

5.25 

188 

5.6 

1977 

4.94 

0.92 

0.96 

0.93 

222.4 

20018 

23 

208 

17.75 

178 

14.1 

1875 

3.81 

0.61 

0.87 

0.99 

157.1 

57090 

38 

209 

14.38 

195 

15.2 

1289 

4.69 

1.74 

0.95 

0.95 

231 

44082 

36 

210 

11.88 

158 

17.1 

1680 

3.2 

1.91 

0.93 

0.97 

167.3 

50152 

43 

211 

5.56 

165 

27.9 

1648 

4.26 

1.13 

0.88 

0.94 

223.9 

72699 

29 

212 

13.44 

166 

26.2 

1422 

4.36 

1.1 

0.98 

1 

175.5 

37145 

22 

213 

12.75 

142 

20.5 

1164 

3.89 

1.36 

0.91 

0.92 

237.6 

40830 

17 

214 

8.63 

150 

18 

1070 

4.92 

1.41 

0.93 

0.98 

166.5 

48201 

31 

215 

10.13 

165 

20.6 

1352 

4.34 

0.6 

0.88 

0.9 

253.6 

42998 

37 

216 

19.13 

159 

29.8 

1547 

3.91 

0.99 

0.95 

0.97 

170 

65328 

26 

217 

14 

157 

25.9 

1688 

4.08 

1.97 

0.89 

0.93 

232.9 

51020 

17 

218 

6.69 

183 

23.5 

1008 

2.68 

1.3 

0.91 

0.99 

189.9 

70748 

20 

219 

10.69 

177 

25.1 

1227 

3.42 

0.46 

0.95 

0.9 

209.5 

47768 

12 

220 

15.06 

197 

25.4 

1383 

3.03 

0.7 

0.91 

0.96 

187.2 

34543 

33 

221 

15.44 

178 

27.2 

1914 

3.48 

1.66 

0.98 

0.92 

214.9 

32375 

36 

222 

8.69 

170 

22 

1453 

4.18 

1.2 

0.86 

0.96 

182.1 

33893 

27 

223 

10.75 

194 

24.7 

1852 

3.75 

0.67 

0.92 

0.93 

242.3 

29773 

27 

224 

13.13 

175 

20.2 

1180 

3.36 

0.1 

0.92 

0.99 

200.1 

47334 

39 

225 

19.5 

173 

27.7 

1133 

2.7 

1.24 

0.94 

0.91 

219.6 

60775 

31 

226 

7.63 

160 

6.4 

2414 

4.49 

1.11 

0.94 

0.94 

198.9 

58824 

42 

227 

10.81 

159 

7.2 

2156 

2.66 

0.36 

0.9 

0.97 

240.3 

64027 

30 

228 

18.56 

153 

6 

2141 

4.06 

0.12 

0.98 

0.9 

172 

31291 

23 

229 

18.94 

145 

4.4 

2938 

3.73 

1.77 

0.88 

0.95 

226.3 

21318 

19 

230 

4.06 

142 

11.4 

2664 

3.96 

1.73 

0.98 

0.94 

163.4 

31508 

36 

231 

5.81 

151 

4.3 

2164 

4.53 

0.01 

0.85 

0.98 

230.2 

30207 

32 

232 

16.94 

156 

14.2 

2383 

2.6 

0.91 

0.96 

0.93 

175.9 

52320 

19 

233 

12.69 

161 

11.6 

2430 

3.71 

1.56 

0.9 

0.97 

204.4 

64461 

12 

234 

4.56 

200 

3.5 

2422 

3.93 

1.55 

0.86 

0.93 

198.1 

73350 

26 

235 

6.81 

193 

15.1 

2234 

3.05 

0.06 

1 

0.97 

236.4 

52537 

12 

236 

15.63 

189 

13.9 

2891 

2.52 

0.63 

0.91 

0.94 

187.6 

40396 

44 

237 

14.5 

173 

13.2 

2250 

3.34 

1.49 

0.96 

0.99 

202.8 

36711 

40 

238 

9.44 

186 

12.1 

2016 

3.98 

1.23 

0.92 

0.91 

176.7 

39746 

14 

239 

9.19 

174 

16.2 

2805 

4 

0.62 

0.97 

0.95 

235.3 

29340 

15 

240 

16.63 

195 

14.8 

2180 

2.64 

0.08 

0.87 

0.95 

182.9 

69447 

31 

241 

18.69 

196 

6.9 

2094 

3.61 

1.46 

0.99 

0.96 

205.6 

69664 

37 

242 

9.38 

148 

20.8 

2047 

3.14 

1.13 

0.97 

0.91 

181.3 

50586 

44 

243 

11.81 

149 

21.2 

2844 

3.59 

0.77 

0.86 

0.95 

215.7 

56439 

41 

244 

13 

147 

24.3 

2719 

4.8 

0.43 

0.98 

0.94 

178.6 

42564 

18 

245 

17.69 

143 

27.8 

2297 

3.65 

1.32 

0.86 

0.99 

249.7 

23920 

11 

246 

4.31 

159 

29.1 

2109 

4.65 

1.27 

0.99 

0.94 

171.2 

20668 

38 

247 

11.19 

143 

25.3 

2305 

4.32 

0.15 

0.89 

0.96 

247.4 

50803 

31 

248 

13.56 

165 

22.6 

2984 

2.58 

0.83 

0.95 

0.92 

161 

45816 

15 

249 

13.94 

159 

20.7 

2695 

4.63 

1.21 

0.91 

1 

206.3 

61643 

21 

250 

9 

199 

29.9 

2516 

3.07 

2 

0.91 

0.93 

166.1 

65111 

27 

251 

4.75 

173 

26.9 

2641 

4.79 

0.88 

0.98 

0.98 

223.5 

72266 

21 

252 

13.38 

176 

24.8 

2656 

2.79 

0.82 

0.9 

0.91 

193.5 

23703 

34 

253 

14.88 

177 

17.7 

2789 

3.16 

1.22 

0.93 

0.99 

210.6 

25438 

41 

254 

6 

180 

23.9 

2406 

2.97 

1.76 

0.92 

0.92 

176.3 

34760 

17 

255 

6.5 

171 

18.5 

2609 

4.2 

0.22 

0.95 

0.97 

216.5 

43648 

27 

256 

18.25 

174 

27.6 

2273 

4.41 

0.25 

0.87 

0.91 

195.4 

70098 

31 

257 

19.19 

191 

28.2 

2828 

3.01 

1.88 

0.93 

0.97 

245.4 

53188 

35 
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APPENDIX  C.  CALCULATING  AN  EXPECTED  VALUE  OF  THE 
RESPONSE  VARIABLE  BASED  ON  REGRESSION  COEFFICIENTS 


The  two  tables  below  show  the  regression  coefficients  and  corresponding  scaled 
estimates  for  a  regression  model  with  five  terms  (plus  the  intercept  term).  If  given  the 
parameter  coefficients  (top  table),  the  formula  for  calculating  an  estimate  of  the  response 
is  simply  obtained  by  adding  the  intercept  term  to  the  sum-product  of  the  regressor  terms 
and  their  corresponding  assigned  value.  For  example,  by  selecting  values  for  the  five 
terms  in  this  model  (within  the  ranges  of  the  original  design)  and  inserting  them  into  the 
formula  below,  an  estimate  of  the  response  is  obtained — in  this  case,  the  proportion  of 
mobile  vehicle  target  kills. 

(-0.424164)  +  (-0.062232  *  Tgt_C_Spd)  +  (0.0048507  *  NEWJmpRad)  + 
(0.9302622  *  NEWPdC)  +  (0.001617828  *  AC_Spd)  +  0.0001395  *  JTAC_SnsrRng) 


If  given  the  scaled  estimates,  a  conversion  must  be  made  to  obtain  the  original 
parameter  estimates.  As  indicated  in  the  bottom  figure,  the  estimates  are  centered  by 
mean  and  scaled  by  range/2.  This  indicates  the  scaled  estimate  was  obtained  by 
multiplying  the  original  coefficient  by  the  value  of  (range  of  factor  setting)/2.  For 
example,  in  the  design,  target  speed  varied  between  4  and  20  m/s.  This  provides  a 
range/2  value  of  (20-4)/2  =  8.  Multiplying  the  original  coefficient  estimate  for  target 
speed  by  8  gives  the  value  of  the  scaled  estimate  (rounding  considered): 

(-  0.062232)  *  8  =  -0.497856. 


Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

-0.424164 

0.42449 

-1.00 

0.3186 

TgtCSpd 

-0.062232 

0.004021 

-15.48 

<.0001 

NEWJmpRad 

0.0048507 

0.002382 

2.04 

0.0428 

NEWPdC 

0.9302622 

0.427881 

2.17 

0.0306 

AC_Spd 

0.0016178 

0.000643 

2.51 

0.0125 

JTACSnsrRng 

0.0001395 

0.000014 

9.75 

<.0001 

Scaled  Estimates 


Prob>|t| 

<.0001 

<.0001 

0.0428 

0.0306 

0.0125 

<.0001 


Continuous  factors  centered  by  mean,  scaled  by  range/2 


Term 

Intercept 

Tgt_C_Spd 

NEWJmpRad 

NEWPdC 

AC_Spd 

JTAC_SnsrRng 


Scaled  Estimate 

0.4830739 

-0.497853 

0.0654849 

0.0697697 

0.0808914 

0.3137775 


I  I  I 


Std  Error 

0.018645 

0.032169 

0.032162 

0.032091 

0.032169 

0.032169 


t  Ratio 

25.91 

-15.48 

2.04 

2.17 

2.51 

9.75 
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APPENDIX  D.  ANNOTATIONS  REGARDING  APPLICABILITY  OF 
MODEL  IN  ADDRESSING  OVERALL  JTEM  REQUIREMENTS 


|  JTEM  Requirement 

Resolution  within  "TheTester" 

Identifier 

Description 

Developer  Comments 

1 

Ability  to  model  JTEM  Test  07  and  JTEM  Test  08 
scenario  and  its  associated  entities.  The  ABM 
should  be  designed  to  model  at  least  the  JTEM  Test 
07  and  JTEM  Test  08  scenarios 

1.1 

Task  Force  organizations  to  support  the  scenario 

P  -  I  don’t  know  what  these  are  yet  to  make  a  further 
assessment.  We  have  Aircraft  (Launch  Platform),  NEW, 
Legacy,  C2ISRNetwork  (assigns  CAS  missions  to  aircraft 
based  on  Target  classification  and  priority  and  weapon- 
target  pairing),  and  WCNetwork  (models  latency  and 
reliability  of  target  updates  getting  from  JTAC  to  NEW) 

1.2 

The  Army's  None-Line-Of-Site  Launch  System 
(NLOS-LS)  platform  with  its  Precision  Air  Munitions 
(PAMs),  including  ability  to  vary  platforms,  PAM 
mix,  and  PAM  range 

N  -  although  if  the  NLOS-LS  can  be  modeled  using  the 
current  Aircraft  Agent,  than  Y.  We’d  need  more 
information  on  what  constitutes  this  package  of  systems. 

1.3 

The  Air  Force's  Networked  Enabled  Weapon  (NEW) 
and  its  delivery  air  platforms,  including  ability  to 
vary  platforms,  NEW  mix,  and  NEW  range 

Y  -  this  was  the  canonical  scenario.  Again,  depends  on 
what  is  included  in  the  "delivery  air  platforms" 

1.4 

Joint  Command  and  Control  (JC2)  entities  with  at 
least  Joint  Tactical  Air  Control  (JTAC)  elements 
(ability  to  vary  types)  and  a  representation  of  the 
Joint  Force  Commander  and  staff.  For  the 
purposes  of  this  ABM  effort,  this  latter 
representation  should  be  kept  simple,  providing  a 
minimal  number  of  representatives  (e.g.,  the  Joint 
Force  Commander,  the  Joint  Force  Air  Component 
Commander,  the  Joint  Force  Land  Component 
Commander,  and  Joint  Force  Effects  Coordination 
Cell)  to  accomplish  JC2  with  respect  to  the  NLOS- 
LS  and  NEW  systems 

N/P  -  not  sure  how  modeling  the  Commanders  will  impact 
the  system.  Will  need  more  information  on  this  to  answer 
fully.  Flowever,  the  current  C2ISRNetworkAgent  only 
models  latency  in  decision  time,  and  the  decision  modeled 
is  the  assignment  of  aircraft  to  a  CAS  mission. 

1.5 

Appropriate  enemy  targets  (including  types/speeds 
of  targets)  to  support  the  test  scenario 

Y.  Stationary  and  mobile  Targets  are  modeled,  with  each 
having  a  target  class  that  can  represent  different  types  of 
targets.  For  mobile  targets,  you  specify  a  path  and  a 
speed.  The  target  class  is  also  linked  to  Pd  and  Pk  for  the 
JTAC/NEW  sensors  and  the  NEW 

1.6 

Ability  of  enemy  to  jam  PAM  and  NEW  systems 

N.  Would  need  to  know  the  different  ways  jamming  effects 
the  systems  ,e.g.,  does  it  change  the  Pd,  add  noise  to  the 
TLE,  other  mechanisms? 

1.7 

Ability  to  model  unit  order  of  battle 

P.  User  can  add  any  number  of  agents  (other  than 
C2ISRNetworkAgent)  by  adding  them  to  the  scenario  file. 
Would  need  more  testing,  though. 

1.8 

Able  to  vary  Blue  and  Red  materiel,  doctrine, 
organizations,  personnel,  and  leadership 

P.  Only  material,  personnel,  and  organizations,  and 
personnel  and  organizations  only  by  number.  Would  need 
to  add  different  types  of  doctrinal  "behaviors"  to  capture 
this  fully.  I  think  this  is  one  area  where  we’d  need  to 
discuss  future  efforts 

1.9 

Clutter  (e.g.,  civilian  personnel,  air  traffic)  to 
evaluate  its  impact  on  system  performance 
operations  (e.g.,  deconfliction  of  air  space  for 
launching  the  NLOS-LS  or  NEW) 

N.  Would  need  more  information  on  what  needs  to  be 
modeled  and  what  interactions  there  are 

1.10 

Ability  to  change  type  of  mission 

P.  Only  one  type  of  mission,  but  can  be  adapted. 

1.11 

Ability  to  add  conceptual  models  of  new  systems 

N.  Certainly  something  to  be  done  for  future  effort.  Would 
need  some  design  work  so  that  conceptual  models  could 
easily  be  "plugged-in" 

1.12 

Ability  to  model  rules  of  engagement  (ROEs) 

N/P.  ROEs,  in  a  limited  sense,  are  "hard-wired".  The  JTAC 
senses  a  Target,  "knows"  it  is  one,  and  then  sends  CAS 
request,  which  then  gets  prosecuted  for  that  target. 

There  is  a  Redirect  by  the  JTAC  if  a  target  of  higher 
priority  presents  itself.  Currently,  the  only  thing  I  would 
call  an  ROE  that  is  implemented. 
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JTEM  Requirement 

Resolution  within  "TheTester" 

Identifier 

Description 

Developer  Comments 

2 

Ability  to  model/assess  JC2 

2.1 

Model/assess  shared  situational  awareness, 
planning,  conduct  of  netcentric  operations,  and  C2 
efficiency 

N.  Would  need  some  kind  of  conceptual  model  to  start 
with.  The  other  question,  is  what  part  of  all  of  THAT 
system  affects  the  system  under  test?  In  the  larger  sense 
of  modeling  systems  of  systems,  I  guess  it  depends  on 
what's  getting  tested  and  where  the  connections  are,  and 
what  of  the  larger  system  affects  those  connections. 

2.2 

Ability  to  conduct  battle  damage  assessment 

N.  BDA  is  not  performed,  nor  information  from  a  BDA 
inserted  into  the  overall  system. 

2.3 

Consider  NATO  Code  of  Best  Practices  for  C2 
Assessment,  published  by  the  Command  and 

Control  Research  Program  (CCRP)  under  the 
auspices  of  the  Assistant  Secretary  of  Defense 
(C3I),  2002 

3 

Ability  to  model/assess  sensor  systems 

N/P.  The  only  set  of  "sensors"  in  the  model  are  those 
attached  to  the  JTAC  and  the  NEW 

3.1 

Unattended  ground  sensors 

3.2 

Sensors  mounted  on  unmanned  ground  and  air 
platforms 

3.3 

Sensors  on  NLOS-LS/PAM  and  NEW  systems 

3.4 

Sensors  for  NLOS  operator  and  JTAC  (including 
target  location  system) 

4 

Ability  to  model/assess  the  Joint  Fires  (including 
Joint  Close  Air  Support)  and  JC2  Doctrine/TTPs  to 
support  the  above  scenarios,  include  ability  to 

4.1 

Vary  doctrine/TTPs 

N 

4.2 

Manage  the  air  space 

N 

4.3 

Model  impacts  on  Circular  Error  Probable  (CEP)  for 
weapon  systems 

P  -  Have  included  TLE  mu/sigma  for  the  NEW 

4.3.1 

NEW  update  rates 

Y 

4.3.2 

NEW-Linkl6  latency 

Y  -  Latency  and  Reliability  of  the  link  are  modeled  using 
uniform  distributions 

4.3.3 

JTAC  effectiveness  (beam  blocked,  beam  split, 
environmental  effects) 

P  -  Only  to  the  extent  that  the  JTAC  is  tracking  the  target 
using  the  sensor,  that  includes  TLE  mu/sigma  factors 

4.3.4 

Relative  CEP 

N  -  Model  uses  bivariate  Normal  TLE  approximation 

4.3.5 

C2  issues 

P  -  Addresses  latency  associated  with  C2  time  to  respond 
to  CAS  request 

5 

Ability  to  model/assess  netcentric  communications 
systems,  including  the  ability  to: 

5.1 

Track  message  traffic 

N 

5.2 

Degrade  system  effectiveness  with  effects  of 
weapons  and  environment 

N 

6 

Ability  to  model  the  environment,  including 

6.1 

Representation  of  the  terrain  where  tests  will  occur 

N  -  How  does  terrain  affect  the  system? 

6.2 

Features  (e.g.,  buildings,  foliage)  and  weather,  or 
at  least  the  effects  that  features  and  weather  have 
on  system  of  systems  performance  and  TTPs 

N/P  -  Currently  modeled  by  varying  the  Pd  and  TLE  of  the 
JTAC  or  NEW 

6.3 

Visibility 

N/P  -  Same  as  6.2 

6.4 

Civilians 

N 

7 

Ability  to  mode/assess  some  human  dimensions: 

7.1 

For  message  traffic  received  by  a  human,  add 
ability  to  adjust  the  understanding  of  the  message 
(this  may  be  as  simple  as  "understand"  and  "does 
not  understand") 

N 

7.2 

Variable  operating  skills  for  the  JTAC  and  NLOS 
personnel 

N/P  -  Throught  the  Pd/TLE  and  IFTU  frequency  currently 

7.3 

Civilian  relationships  to  Blue  and  Red 

N  -  Neutrals  currently  not  modeled 

7.4 

Ability  to  affect  human  performance  (e.g., 
information/message  overload  slows  down  decision 
processes  or  causes  an  action  to  be  missed 
altogether) 

N/P  -  Currently,  only  in  C2  decision  latency 
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